{"slug": "the-cheapest-llm-call-is-the-one-you-don-t-make-a-caching-layer-that-actually", "title": "The cheapest LLM call is the one you don't make: a caching layer that actually pays off", "summary": "A developer describes how adding a caching layer to LLM API calls significantly reduced costs beyond multi-provider routing. By hashing full requests and using semantic caching for similar prompts, the team avoided unnecessary model calls, achieving substantial savings on high-traffic endpoints.", "body_md": "The cheapest LLM call is the one you don't make: a caching layer that actually pays off\n\n*In the last post I wrote about routing across providers to cut our bill ~40%. Caching was the second lever — and honestly the more underrated one. Here's what we learned shipping it.*\n\nRouting gets most of the attention because it's sexy: traffic dancing across providers, failover kicking in, dashboards lighting up. But the single biggest cost lever we pulled after routing wasn't smarter routing. It was not calling the model at all.\n\nWhen people talk about LLM cost, they picture the per-token price. That's the wrong unit. The question is how many of your calls are *genuinely new information* versus repeats wearing a costume.\n\nWe were shocked at the overlap. Once we started measuring, a large share of production traffic was re-asking near-identical things:\n\nNone of that needs a fresh model call. It needs a cache with a brain.\n\nHash the full request (system + messages + params). If you've seen it, return the stored completion. Obvious, but most teams skip it because \"our prompts are dynamic.\" They usually aren't *that* dynamic.\n\n``` python\nimport hashlib, json\n\ndef cache_key(req):\n    return hashlib.sha256(json.dumps(req, sort_keys=True).encode()).hexdigest()\n\ndef complete(req):\n    k = cache_key(req)\n    hit = store.get(k)\n    if hit:\n        return hit  # zero tokens spent\n    out = model_call(req)\n    store.set(k, out, ttl=300)\n    return out\n```\n\nThis alone killed a chunk of bill on our highest-traffic endpoints.\n\nExact matching misses the real win: *similar* prompts returning *similar* answers. Embed the user turn, store embeddings in a vector index, and on each request check for a neighbor above a similarity threshold (we use ~0.92). If found, reuse the prior completion.\n\nThe catch: semantic caching is only safe for deterministic-ish tasks (classifications, extractions, stable Q&A). Don't cache creative generation — you'll serve stale voices. We scope it tightly and it still covers a surprising volume.\n\nA lot of \"LLM calls\" are actually deterministic work wrapped in a prompt: parsing, normalization, format conversion. We moved those to pure functions computed once and reused. It's not even a model cache — it's just not pretending the model is needed.\n\nNone of this is exotic. It's the same caching discipline people have applied to databases for decades, applied to model calls where the per-hit savings are bigger.\n\nRouting moves traffic to the cheapest healthy provider ([how we cut the bill with routing](https://dev.to/xuanyi/how-we-cut-our-llm-bill-40-with-multi-provider-routing-44l7)). A circuit breaker keeps a flaky provider from turning an outage into a bill explosion ([the pattern we use](https://majiafan.hashnode.dev/a-practical-circuit-breaker-for-llm-api-calls-in-production)). Caching is the layer underneath both: the call you skip is the call you never have to route or protect.\n\nGetting reliable, affordable model access set up for a team has its own headaches — provider quotas, region limits, payment friction. If any of that sounds familiar, I'm happy to compare notes. Find me here or DM me; no pitch, just war stories.", "url": "https://wpnews.pro/news/the-cheapest-llm-call-is-the-one-you-don-t-make-a-caching-layer-that-actually", "canonical_source": "https://dev.to/xuanyi/the-cheapest-llm-call-is-the-one-you-dont-make-a-caching-layer-that-actually-pays-off-59e", "published_at": "2026-08-19 02:34:43+00:00", "updated_at": "2026-08-19 02:42:13.394982+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/the-cheapest-llm-call-is-the-one-you-don-t-make-a-caching-layer-that-actually", "markdown": "https://wpnews.pro/news/the-cheapest-llm-call-is-the-one-you-don-t-make-a-caching-layer-that-actually.md", "text": "https://wpnews.pro/news/the-cheapest-llm-call-is-the-one-you-don-t-make-a-caching-layer-that-actually.txt", "jsonld": "https://wpnews.pro/news/the-cheapest-llm-call-is-the-one-you-don-t-make-a-caching-layer-that-actually.jsonld"}}