# AI SaaS: Cut the Bill Before You Buy More GPUs

> Source: <https://dev.to/alok1663/ai-saas-cut-the-bill-before-you-buy-more-gpus-1bn2>
> Published: 2026-09-25 11:04:24+00:00

The board deck said to scale inference. The Grafana dashboard showed a different bottleneck: Postgres CPU sat at 78%, and `GET /v1/models` handled 40,000 requests per minute, all returning identical JSON.

This pattern shows up repeatedly at AI startups that already solved streaming completions. The expensive GPU fleet gets all the attention, but boring HTTP traffic often breaks the infrastructure first. Model lists, pricing sheets, feature flags, public prompt templates, and dashboard summaries hammer the primary database on every page load and client poll.

Here is how to offload that read traffic at the edge without touching your streaming chat pipeline.

When we examined the telemetry from three production teams (a B2B coding copilot, an automated legal assistant, and a customer support agent platform), the request distribution looked almost identical:

| Traffic type | Share of requests | Cost profile | 
|---|---|---|
| Streaming completions ( `POST` ) | 8% to 15% | High per request (GPU time, token counts) | 
| Embedding generation jobs | 5% to 15% | Moderate (vector API batches) | 
| **Read APIs** (models, configs, pricing, prompt libraries) | **70% to 85%** | Small per request, massive in aggregate | 

A single request to `GET /v1/models` takes negligible CPU. But 40,000 requests per minute against an ORM query that joins model variants, token pricing tiers, context windows, and provider availability will saturate a database connection pool.

When your web frontend, mobile client, and VS Code extension poll that endpoint on every reload, you pay a heavy database tax for static data.

`/v1/models` hits the database so hard
In most codebases, `GET /v1/models` is not a flat JSON file. It is an ORM call that looks like this:

``` python
# FastAPI + SQLAlchemy example
@app.get("/v1/models")
async def list_models(db: AsyncSession = Depends(get_db)):
    query = (
        select(Model)
        .options(
            joinedload(Model.pricing_tiers),
            joinedload(Model.provider_status),
            joinedload(Model.supported_parameters)
        )
        .where(Model.is_active == True)
        .order_by(Model.display_order)
    )
    result = await db.execute(query)
    models = result.unique().scalars().all()
    return {"data": [m.to_dict() for m in models]}
```

Every incoming client request does three things:

`LEFT OUTER JOIN` clauses.
Under burst traffic, connection pools fill up. New chat sessions fail to acquire database connections to save conversation state, leading to 500 errors during peak user activity. The GPU cluster is completely idle, but the application fails because the catalog API exhausted the database pool.

[ApexCache](https://getapexcache.com) only caches HTTP `GET` and `HEAD` methods. Every other method passes directly through to your origin servers.

This aligns cleanly with standard AI API design:

`GET /v1/models`: Cache for 60 to 300 seconds. Add a surrogate tag (` Cache-Tag: models`).` GET /v1/pricing`: Cache for 300 to 1800 seconds.` GET /v1/prompts/public`: Cache for 60 to 300 seconds with query string preservation.`GET /v1/docs` or `GET /openapi.json`: Cache for 3600 seconds.` POST /v1/chat/completions`: Streaming tokens and user prompts must never touch an edge cache.`POST /v1/embeddings`: Unique text embeddings belong on the origin.
You are not caching the AI engine. You are caching the menu that every user reads before placing an order.

To keep data fresh without waiting for long TTL timeouts, return a `Cache-Tag` header from your origin server. When your marketing or engineering team updates a model parameter or pricing tier, invalidate that specific tag:

``` js
// Express / Node.js origin response
app.get("/v1/models", async (req, res) => {
  const models = await fetchModelsFromPostgres();

  // Instruct edge proxy to cache and tag this response
  res.setHeader("Cache-Control", "public, s-maxage=300");
  res.setHeader("Cache-Tag", "models,catalog");

  res.json({ object: "list", data: models });
});
```

When you deploy a new model variant or adjust rate limits, trigger a tag purge through the ApexCache API:

```
curl -X POST "https://api.getapexcache.com/api/v1/cache/invalidate" \
  -H "Authorization: Bearer $APEXCACHE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"tags":["models"]}'
```

Every edge node drops the cached `/v1/models` payload in under 10 milliseconds. The next request fetches fresh data from your origin and repopulates edge memory.

Consider an AI SaaS application handling 30 million read API requests each month:

By reducing read QPS against the primary database by 88%, the team avoided adding a second RDS read replica ($260/month) and delayed upgrading the primary instance to `db.r6g.2xlarge` ($1,040/month).

The infrastructure savings came from eliminating redundant database queries, not from tweaking LLM prompt tokens.

You do not need to create a separate domain for caching. You can point your existing API domain (`api.yourproduct.com`) or specific path prefixes to ApexCache:

`cname.getapexcache.com`) and the verification TXT record in your DNS provider.`/v1/models*` with a TTL of 120 seconds.`curl`:

```
curl -sI "https://api.yourproduct.com/v1/models" | grep -i x-apexcache
```

On the first request, the response header reads:

```
X-ApexCache-Status: MISS
```

On the second request, the response returns directly from memory:

```
X-ApexCache-Status: HIT
Age: 4
```

Keep these three boundaries in mind when introducing edge caching to an AI API:

`/v1/pricing`) rather than broad wildcards (`/*`), so that streaming completions (`/v1/chat/completions`) never encounter cache inspection overhead.
Edge caching does not reduce your OpenAI or Anthropic invoice. It fixes the infrastructure tax around your product so your database stays online when traffic spikes.

If your database CPU spikes while your GPU servers remain underutilized:

`k6` or `hey` and monitor your database CPU before and after caching.
Docs: [getapexcache.com/docs](https://getapexcache.com/docs) · Contact: [getapexcache.com/contact](https://getapexcache.com/contact)

*I work on ApexCache. Numbers in this article are based on production benchmarks. Run your own load test before sizing production infrastructure.*