AI SaaS: Cut the Bill Before You Buy More GPUs

A developer analyzed production telemetry from three AI teams and found that read APIs such as GET /v1/models, pricing, and public prompt endpoints account for 70% to 85% of requests, saturating Postgres connection pools while GPU capacity sits idle. The writeup recommends edge-caching only GET and HEAD responses with short TTLs and Cache-Tag headers for targeted invalidation, while leaving streaming completions and embedding POSTs uncached.

The board deck said to scale inference. The Grafana dashboard showed a different bottleneck: Postgres CPU sat at 78%, and GET /v1/models handled 40,000 requests per minute, all returning identical JSON. This pattern shows up repeatedly at AI startups that already solved streaming completions. The expensive GPU fleet gets all the attention, but boring HTTP traffic often breaks the infrastructure first. Model lists, pricing sheets, feature flags, public prompt templates, and dashboard summaries hammer the primary database on every page load and client poll. Here is how to offload that read traffic at the edge without touching your streaming chat pipeline. When we examined the telemetry from three production teams a B2B coding copilot, an automated legal assistant, and a customer support agent platform , the request distribution looked almost identical: | Traffic type | Share of requests | Cost profile | |---|---|---| | Streaming completions POST | 8% to 15% | High per request GPU time, token counts | | Embedding generation jobs | 5% to 15% | Moderate vector API batches | | Read APIs models, configs, pricing, prompt libraries | 70% to 85% | Small per request, massive in aggregate | A single request to GET /v1/models takes negligible CPU. But 40,000 requests per minute against an ORM query that joins model variants, token pricing tiers, context windows, and provider availability will saturate a database connection pool. When your web frontend, mobile client, and VS Code extension poll that endpoint on every reload, you pay a heavy database tax for static data. /v1/models hits the database so hard In most codebases, GET /v1/models is not a flat JSON file. It is an ORM call that looks like this: python FastAPI + SQLAlchemy example @app.get "/v1/models" async def list models db: AsyncSession = Depends get db : query = select Model .options joinedload Model.pricing tiers , joinedload Model.provider status , joinedload Model.supported parameters .where Model.is active == True .order by Model.display order result = await db.execute query models = result.unique .scalars .all return {"data": m.to dict for m in models } Every incoming client request does three things: LEFT OUTER JOIN clauses. Under burst traffic, connection pools fill up. New chat sessions fail to acquire database connections to save conversation state, leading to 500 errors during peak user activity. The GPU cluster is completely idle, but the application fails because the catalog API exhausted the database pool. ApexCache https://getapexcache.com only caches HTTP GET and HEAD methods. Every other method passes directly through to your origin servers. This aligns cleanly with standard AI API design: GET /v1/models : Cache for 60 to 300 seconds. Add a surrogate tag Cache-Tag: models . GET /v1/pricing : Cache for 300 to 1800 seconds. GET /v1/prompts/public : Cache for 60 to 300 seconds with query string preservation. GET /v1/docs or GET /openapi.json : Cache for 3600 seconds. POST /v1/chat/completions : Streaming tokens and user prompts must never touch an edge cache. POST /v1/embeddings : Unique text embeddings belong on the origin. You are not caching the AI engine. You are caching the menu that every user reads before placing an order. To keep data fresh without waiting for long TTL timeouts, return a Cache-Tag header from your origin server. When your marketing or engineering team updates a model parameter or pricing tier, invalidate that specific tag: js // Express / Node.js origin response app.get "/v1/models", async req, res = { const models = await fetchModelsFromPostgres ; // Instruct edge proxy to cache and tag this response res.setHeader "Cache-Control", "public, s-maxage=300" ; res.setHeader "Cache-Tag", "models,catalog" ; res.json { object: "list", data: models } ; } ; When you deploy a new model variant or adjust rate limits, trigger a tag purge through the ApexCache API: curl -X POST "https://api.getapexcache.com/api/v1/cache/invalidate" \ -H "Authorization: Bearer $APEXCACHE API KEY" \ -H "Content-Type: application/json" \ -d '{"tags": "models" }' Every edge node drops the cached /v1/models payload in under 10 milliseconds. The next request fetches fresh data from your origin and repopulates edge memory. Consider an AI SaaS application handling 30 million read API requests each month: By reducing read QPS against the primary database by 88%, the team avoided adding a second RDS read replica $260/month and delayed upgrading the primary instance to db.r6g.2xlarge $1,040/month . The infrastructure savings came from eliminating redundant database queries, not from tweaking LLM prompt tokens. You do not need to create a separate domain for caching. You can point your existing API domain api.yourproduct.com or specific path prefixes to ApexCache: cname.getapexcache.com and the verification TXT record in your DNS provider. /v1/models with a TTL of 120 seconds. curl : curl -sI "https://api.yourproduct.com/v1/models" | grep -i x-apexcache On the first request, the response header reads: X-ApexCache-Status: MISS On the second request, the response returns directly from memory: X-ApexCache-Status: HIT Age: 4 Keep these three boundaries in mind when introducing edge caching to an AI API: /v1/pricing rather than broad wildcards / , so that streaming completions /v1/chat/completions never encounter cache inspection overhead. Edge caching does not reduce your OpenAI or Anthropic invoice. It fixes the infrastructure tax around your product so your database stays online when traffic spikes. If your database CPU spikes while your GPU servers remain underutilized: k6 or hey and monitor your database CPU before and after caching. Docs: getapexcache.com/docs https://getapexcache.com/docs · Contact: getapexcache.com/contact https://getapexcache.com/contact I work on ApexCache. Numbers in this article are based on production benchmarks. Run your own load test before sizing production infrastructure.