Prompt Caching, Explained: How to Cut Your LLM Bill by 70-90% (With Real Math)

In my last post I broke down how LLMs count tokens and why your bill is decided at the tokenizer. A lot of the follow-up questions were the same: "Okay — so how do I actually pay less?"

This is the answer, and it's the single highest-leverage cost lever available on Claude, GPT, and Gemini today: prompt caching. It requires no model change, no quality tradeoff, and often just a few lines of code. Done well, it cuts input token costs by 70-90% on typical production workloads. Done poorly, it does nothing — or quietly makes things worse.

Let's break down how it works, the real cost math, and the mistakes that leave most of the savings on the table.

When a model processes your prompt, it computes attention key-value (KV) tensors for every token. That's real compute, and you pay for it on every request.

Here's the thing: most production prompts are mostly repetition. The system prompt, the tool definitions, the few-shot examples, the big RAG document — that prefix is identical on call after call. Only the last bit (the user's actual message) changes.

Prompt caching stores the already-computed KV tensors for that repeated prefix server-side. When your next request starts with the same prefix, the model skips recomputing it and loads the cached result instead — and bills those cached tokens at a steep discount.

The mental model: you're paying full price to process your system prompt once, then paying a fraction of that to reuse it for the next N requests.

Caching works on prefixes. The provider can only reuse everything up to the first point where your prompt differs from last time.

That single fact dictates everything:

Put stable content at the front. Put anything that changes at the very end.

Structure your prompt like this:

[ system prompt        ]  ← static, cache this
[ tool definitions     ]  ← static, cache this
[ few-shot examples    ]  ← static, cache this
[ retrieved documents  ]  ← semi-static
--------------------------------------------------
[ conversation history ]  ← changes
[ user's new message   ]  ← changes every time

The instant something dynamic leaks into the front — a timestamp, a request ID, a randomly ordered tool list, even an inconsistent trailing newline — the cache invalidates for every token after it. This is the number-one reason teams see "90% off" pricing advertised and a 20% hit rate in reality.

The three big providers all discount cached input by roughly 90%, but the mechanics and fine print differ enough to change your architecture.

You opt in by marking stable blocks with cache_control

(up to 4 breakpoints). Claude splits your bill into:

Default TTL is 5 minutes, and every read refreshes the timer. There's also an automatic mode now, but explicit breakpoints give you the most control. The write surcharge typically pays for itself on the first hit.

Caching happens automatically on supported models — no markers, no cache objects. The catch is a hard 1,024-token minimum prefix: a 900-token system prompt will never cache, no matter how consistent it is. Retention runs from a few minutes of inactivity up to 24 hours on newer models. Simplest to adopt, least to tune.

Gemini offers implicit caching (automatic, ~10% read rate) and explicit caching (you create a named cache object and reference it by ID). The gotchas:

Gemini caching rewards big-document workloads, not typical short prompt engineering.

(Mechanics and rates shift often — confirm on each provider's current docs before budgeting.)

Let's make it concrete. Say you have a 5,000-token system prompt reused across 10,000 requests/day, on a model at $3 / 1M input with a cache read at $0.30 / 1M (90% off) and a 5-minute write at $3.75 / 1M.

Every request pays full price for those 5,000 tokens:

10,000 req × 5,000 tokens = 50,000,000 input tokens/day
50,000,000 / 1,000,000 × $3 = $150/day  → ~$4,500/month

(That's just the cached-portion input — actual bills add the dynamic tail and output.)

Assume the cache is written fresh a handful of times a day as it expires — say ~50 writes — and everything else is a read:

Writes:  50 × 5,000 / 1,000,000 × $3.75      ≈ $0.94/day
Reads:   9,950 × 5,000 / 1,000,000 × $0.30   ≈ $14.93/day
------------------------------------------------------------
Total ≈ $15.87/day  → ~$476/month

That's the repeated-prefix cost dropping from ~$4,500 to ~$476 a month — about 89% off — by making one part of your prompt cacheable. The often-cited real-world version of this is an agent that went from $720/month to $72/month by adding three cache breakpoints.

The savings scale with two things: how big your static prefix is and how often you reuse it within the TTL. Big system prompt + high request rate = enormous savings. Tiny prompt + sporadic traffic = little to none.

Caching fails quietly. You still get correct responses — you just don't get the discount, and nothing errors out to tell you. Watch for these:

Don't trust the marketing — trust the usage metadata. Every provider exposes cache stats (field names differ):

cache_read_input_tokens

, cache_creation_input_tokens

cached_tokens

A quick health check:

hit_rate ≈ cached_input_tokens / total_cache_eligible_input_tokens

If that number is consistently near zero, dynamic content has leaked into your prefix. A sudden drop usually means something changed in the reusable part of your prompt.

Caching won't fix a badly chosen model or a bloated context. But for the very common case of "big stable prompt, called a lot," it's the closest thing to free money in the LLM stack. Most teams are leaving 70-90% of it on the table.

Have you shipped prompt caching in production? What was your real hit rate — and what broke it? I'd love to hear the war stories in the comments.