{"slug": "prompt-caching-explained-how-to-cut-your-llm-bill-by-70-90-with-real-math", "title": "Prompt Caching, Explained: How to Cut Your LLM Bill by 70-90% (With Real Math)", "summary": "Prompt caching can reduce LLM input token costs by 70-90% on production workloads by storing computed attention key-value tensors for repeated prompt prefixes. The technique requires no model changes and works on Claude, GPT, and Gemini, but effectiveness depends on prompt structure and provider-specific mechanics. Developers must place stable content at the front and dynamic content at the end to avoid cache invalidation.", "body_md": "In my last post I broke down how LLMs count tokens and why your bill is decided at the tokenizer. A lot of the follow-up questions were the same: *\"Okay — so how do I actually pay less?\"*\n\nThis is the answer, and it's the single highest-leverage cost lever available on Claude, GPT, and Gemini today: **prompt caching.** It requires no model change, no quality tradeoff, and often just a few lines of code. Done well, it cuts input token costs by 70-90% on typical production workloads. Done poorly, it does nothing — or quietly makes things worse.\n\nLet's break down how it works, the real cost math, and the mistakes that leave most of the savings on the table.\n\nWhen a model processes your prompt, it computes attention key-value (KV) tensors for every token. That's real compute, and you pay for it on every request.\n\nHere's the thing: most production prompts are **mostly repetition**. The system prompt, the tool definitions, the few-shot examples, the big RAG document — that prefix is identical on call after call. Only the last bit (the user's actual message) changes.\n\nPrompt caching stores the already-computed KV tensors for that repeated prefix server-side. When your next request starts with the same prefix, the model **skips recomputing it** and loads the cached result instead — and bills those cached tokens at a steep discount.\n\nThe mental model: you're paying full price to process your system prompt *once*, then paying a fraction of that to reuse it for the next N requests.\n\nCaching works on **prefixes**. The provider can only reuse everything up to the first point where your prompt differs from last time.\n\nThat single fact dictates everything:\n\nPut stable content at the front. Put anything that changes at the very end.\n\nStructure your prompt like this:\n\n```\n[ system prompt        ]  ← static, cache this\n[ tool definitions     ]  ← static, cache this\n[ few-shot examples    ]  ← static, cache this\n[ retrieved documents  ]  ← semi-static\n--------------------------------------------------\n[ conversation history ]  ← changes\n[ user's new message   ]  ← changes every time\n```\n\nThe instant something dynamic leaks into the front — a timestamp, a request ID, a randomly ordered tool list, even an inconsistent trailing newline — the cache invalidates for *every token after it*. This is the number-one reason teams see \"90% off\" pricing advertised and a 20% hit rate in reality.\n\nThe three big providers all discount cached input by roughly 90%, but the mechanics and fine print differ enough to change your architecture.\n\nYou opt in by marking stable blocks with `cache_control`\n\n(up to 4 breakpoints). Claude splits your bill into:\n\nDefault TTL is 5 minutes, and every read refreshes the timer. There's also an automatic mode now, but explicit breakpoints give you the most control. The write surcharge typically pays for itself on the *first* hit.\n\nCaching happens automatically on supported models — no markers, no cache objects. The catch is a hard **1,024-token minimum prefix**: a 900-token system prompt will *never* cache, no matter how consistent it is. Retention runs from a few minutes of inactivity up to 24 hours on newer models. Simplest to adopt, least to tune.\n\nGemini offers implicit caching (automatic, ~10% read rate) and explicit caching (you create a named cache object and reference it by ID). The gotchas:\n\nGemini caching rewards big-document workloads, not typical short prompt engineering.\n\n| Claude | GPT | Gemini | |\n|---|---|---|---|\n| Mode | Explicit (`cache_control` ) + auto |\nAutomatic | Implicit + explicit |\n| Read discount | ~90% off | ~50-90% off | ~90% off |\n| Write cost | 1.25×-2× base | none extra | storage/hour |\n| Min prefix | small | 1,024 tokens | very large |\n| Default TTL | 5 min (refreshes) | up to 24h | configurable |\n\n*(Mechanics and rates shift often — confirm on each provider's current docs before budgeting.)*\n\nLet's make it concrete. Say you have a **5,000-token system prompt** reused across **10,000 requests/day**, on a model at **$3 / 1M input** with a cache read at **$0.30 / 1M** (90% off) and a 5-minute write at **$3.75 / 1M**.\n\nEvery request pays full price for those 5,000 tokens:\n\n```\n10,000 req × 5,000 tokens = 50,000,000 input tokens/day\n50,000,000 / 1,000,000 × $3 = $150/day  → ~$4,500/month\n```\n\n*(That's just the cached-portion input — actual bills add the dynamic tail and output.)*\n\nAssume the cache is written fresh a handful of times a day as it expires — say ~50 writes — and everything else is a read:\n\n```\nWrites:  50 × 5,000 / 1,000,000 × $3.75      ≈ $0.94/day\nReads:   9,950 × 5,000 / 1,000,000 × $0.30   ≈ $14.93/day\n------------------------------------------------------------\nTotal ≈ $15.87/day  → ~$476/month\n```\n\nThat's the repeated-prefix cost dropping from **~$4,500 to ~$476 a month — about 89% off** — by making one part of your prompt cacheable. The often-cited real-world version of this is an agent that went from **$720/month to $72/month** by adding three cache breakpoints.\n\nThe savings scale with two things: **how big your static prefix is** and **how often you reuse it within the TTL**. Big system prompt + high request rate = enormous savings. Tiny prompt + sporadic traffic = little to none.\n\nCaching fails quietly. You still get correct responses — you just don't get the discount, and nothing errors out to tell you. Watch for these:\n\nDon't trust the marketing — trust the usage metadata. Every provider exposes cache stats (field names differ):\n\n`cache_read_input_tokens`\n\n, `cache_creation_input_tokens`\n\n`cached_tokens`\n\nA quick health check:\n\n```\nhit_rate ≈ cached_input_tokens / total_cache_eligible_input_tokens\n```\n\nIf that number is consistently near zero, dynamic content has leaked into your prefix. A sudden drop usually means something changed in the reusable part of your prompt.\n\nCaching won't fix a badly chosen model or a bloated context. But for the very common case of \"big stable prompt, called a lot,\" it's the closest thing to free money in the LLM stack. Most teams are leaving 70-90% of it on the table.\n\n*Have you shipped prompt caching in production? What was your real hit rate — and what broke it? I'd love to hear the war stories in the comments.*", "url": "https://wpnews.pro/news/prompt-caching-explained-how-to-cut-your-llm-bill-by-70-90-with-real-math", "canonical_source": "https://dev.to/james_anderson_h/prompt-caching-explained-how-to-cut-your-llm-bill-by-70-90-with-real-math-3cna", "published_at": "2026-08-19 11:07:02+00:00", "updated_at": "2026-08-19 11:42:22.022342+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Claude", "GPT", "Gemini"], "alternates": {"html": "https://wpnews.pro/news/prompt-caching-explained-how-to-cut-your-llm-bill-by-70-90-with-real-math", "markdown": "https://wpnews.pro/news/prompt-caching-explained-how-to-cut-your-llm-bill-by-70-90-with-real-math.md", "text": "https://wpnews.pro/news/prompt-caching-explained-how-to-cut-your-llm-bill-by-70-90-with-real-math.txt", "jsonld": "https://wpnews.pro/news/prompt-caching-explained-how-to-cut-your-llm-bill-by-70-90-with-real-math.jsonld"}}