The cheapest LLM call is the one you don't make: a caching layer that actually pays off
In the last post I wrote about routing across providers to cut our bill ~40%. Caching was the second lever — and honestly the more underrated one. Here's what we learned shipping it.
Routing gets most of the attention because it's sexy: traffic dancing across providers, failover kicking in, dashboards lighting up. But the single biggest cost lever we pulled after routing wasn't smarter routing. It was not calling the model at all.
When people talk about LLM cost, they picture the per-token price. That's the wrong unit. The question is how many of your calls are genuinely new information versus repeats wearing a costume.
We were shocked at the overlap. Once we started measuring, a large share of production traffic was re-asking near-identical things:
None of that needs a fresh model call. It needs a cache with a brain.
Hash the full request (system + messages + params). If you've seen it, return the stored completion. Obvious, but most teams skip it because "our prompts are dynamic." They usually aren't that dynamic.
import hashlib, json
def cache_key(req):
return hashlib.sha256(json.dumps(req, sort_keys=True).encode()).hexdigest()
def complete(req):
k = cache_key(req)
hit = store.get(k)
if hit:
return hit # zero tokens spent
out = model_call(req)
store.set(k, out, ttl=300)
return out
This alone killed a chunk of bill on our highest-traffic endpoints.
Exact matching misses the real win: similar prompts returning similar answers. Embed the user turn, store embeddings in a vector index, and on each request check for a neighbor above a similarity threshold (we use ~0.92). If found, reuse the prior completion.
The catch: semantic caching is only safe for deterministic-ish tasks (classifications, extractions, stable Q&A). Don't cache creative generation — you'll serve stale voices. We scope it tightly and it still covers a surprising volume.
A lot of "LLM calls" are actually deterministic work wrapped in a prompt: parsing, normalization, format conversion. We moved those to pure functions computed once and reused. It's not even a model cache — it's just not pretending the model is needed.
None of this is exotic. It's the same caching discipline people have applied to databases for decades, applied to model calls where the per-hit savings are bigger.
Routing moves traffic to the cheapest healthy provider (how we cut the bill with routing). A circuit breaker keeps a flaky provider from turning an outage into a bill explosion (the pattern we use). Caching is the layer underneath both: the call you skip is the call you never have to route or protect.
Getting reliable, affordable model access set up for a team has its own headaches — provider quotas, region limits, payment friction. If any of that sounds familiar, I'm happy to compare notes. Find me here or DM me; no pitch, just war stories.