Cutting LLM inference costs by 36% with prompt caching

← All posts Prompt caching cut our LLM inference cost by 36% in production. How prompt caching works, where to put cache breakpoints in a long system prompt and tool schema, and the mistakes that quietly destroy your cache hit rate.

We had recently built a human-in-the-loop ReAct agent for a gaming company, handling VIP customer support. VIP support requires building elaborate context that includes procedure documents, the player preferences, account details, lifetime value, which deals they're eligible for, etc. That's a lot of context per conversation, and we had a suspicion it's going to get costly fast.

But since early optimization is the oponent of delivery, we decided to build first and optimize later. Once the agent hit production, we looked at the bill.

As expected, cost per interaction came in slightly above what we'd modeled. It wasn't a crisis, but it WAS enough to become one at the volume they wanted to reach. So we started where you should always start, with the prompt.

Two passes later the bill was down 36%, with no measurable change in output quality.

Here's what we did, in order, and what didn't work, so you can do it yourself:

What you're actually paying for #

The important thing to understand about how [ReAct agents](/post/how-to-build-react-agents-in-2026) work is that **you are re-sending the entire prompt on every agent loop.**

From the user's perspective an "interaction" is one message. But to the API it's a loop. The agent reads the context, decides to call a tool, gets a result, then reads everything again plus that result, and decides what to do next. A conversation with two tool calls is at least three model requests. Request three carries the full system prompt, the full tool schema, the full player profile, and the whole conversation so far.

That prompt is billed at full input price every time. It doesn't matter that it might be byte-for-byte identical to what you sent nine seconds ago.

Our prompt had a natural three-tier structure, which turned out to matter a lot:

The rough token split looked like this:

Tier 1 is 76% of the input on the first request of a ticket. It's the same bytes as the last ticket's tier 1, and the one before that, and every ticket we will ever run. We were paying full price for it a few hundred thousand times a day.

Pass one: make the prompt smaller #

Before caching anything, shrink it. Caching a bloated prompt just stores the bloat more cheaply.

Three things help:

Don't repeat yourself. This is the big one, and it's easy to miss, because prompts don't get written so much as pile up. Someone hits an edge case and adds a rule. Someone else hits a similar case in a different section and adds a similar rule, worded slightly differently. Six months later the same instruction sits in three places and two of them disagree. That's not just wasted tokens. It's a behavior bug you haven't noticed.Say it once, plainly. Models love to pad prompts, so if you've been iterating with a model's help, yours is padded. "It is absolutely critical that you must always ensure that you verify" is nine tokens of nothing.Prefer rules over examples. A worked example is expensive and narrow. A stated rule is cheap and general. Keep examples only where the output format is tricky and a rule can't pin it down.

Our approach was deliberately lazy: let the model compact it first, then review the important logic by hand. Feed the prompt back with "remove redundancy and simplify phrasing, change no behavior," then read the diff carefully. The model will happily delete a load-bearing constraint that looked like filler. The mechanical part is automatable. The judgment isn't.

That got us 15% fewer tokens, on prompts we already thought were tight. If you've never done this pass, expect more.

This is the best kind of saving, because it's free and it shrinks the base that every later optimization works on.

What is prompt caching? #

Prompt caching means the provider stores the bits of prompt it already processed and lets you reuse it almost for free. If a request starts with exactly the same text as an earlier one, that part is served from cache: you pay about 10% of the normal input price for those tokens, and the model doesn't re-read them, so the first token comes back faster too.

So caching only pays on the parts that never change: your system prompt, your tool schemas, your examples, the long document you paste in every time. Those are the cheapest LLM tokens you can buy.

This feature is available in one version or another with all LLM proviers: Anthropic, OpenAI, Gemini, Amazon Bedrock and most self hosted stacks as well. So no matter where you are, you can benefit from this: identical bytes, from the very start of the prompt = savings.

How prompt caching is priced #

You can't reason about the rest of this without the pricing model. The request is assembled in a fixed order — tools

→ system

→ messages

— and the hash runs from the start of that assembly up to your breakpoint. That's why the diagram above is drawn the way it is: anything volatile above the marker invalidates everything below it.

The economics, for Claude on Bedrock:

| cost, relative to normal input | |

|---|---|
cache write (5-minute TTL) | 1.25× |
cache write (1-hour TTL) | 2× |

cache read | ~0.1× | | no cache | 1× |

Reads are nearly free, writes cost extra. So caching is a bet: pay 25% more once, to pay 90% less every time after. And like any bet you can lose it — write an entry that never gets read and you've paid 25% more for that request.

The break-even, for a segment read times after one write:

So on paper you break even after barely more than one reuse. Remember that number. It's about to mislead us.

Two more mechanics worth knowing:

Often, there's a minimum cacheable prefix. Model-dependent, somewhere in the 512–4096 token range. Below it nothing caches — no error, just a cache-write count of zero while you wonder why your dashboard never moves.On Bedrock you place the breakpoints yourself. No automatic mode: at most four, and you decide where they go. Arguably a good thing, since it forces you to think about your tier boundaries.

Pass two: caching, and the tier that didn't pay #

The obvious move was to cache tier 1 and tier 2: one breakpoint at the end of the fixed block, one at the end of the ticket context, tier 3 below the last marker where volatile content belongs. We expected close to half the remaining input cost to disappear.

After a few days of production traffic, tier 1 was working as intended. Tier 2 was doing almost nothing.

Go back to that break-even of . The number is right, but it answers the wrong question. Not "does this segment break even?" but "how many times does this cache entry get read before it expires?"

For tier 1: a lot. One write, then every request from every ticket for the next five minutes reads it. Over a busy hour that's thousands of reads per write, and the write premium disappears into the noise.

For tier 2: **as many requests as this one ticket makes.** The entry is written when the ticket starts and is worthless the moment it ends, because the next ticket is a different player. And our agent usually made only **1–2 tool calls per ticket** — 2–3 model requests, so 1–2 reads per write. That's the number we hadn't taken seriously enough.

The saving is real; 33% isn't nothing. But tier 2 is only about a fifth of the input, so we were claiming a third of a fifth — call it 6% on a good day. Against that: more complexity, one of four breakpoints spent, and one more thing that breaks silently when someone reorders a field in the player profile serializer.

We kept the tier 1 breakpoint and dropped the tier 2 one. Pass two came out at 25% off what pass one left us.

That's where the 36% comes from. Compounding percentages is where cost-saving posts get sloppy, so here it is in full:

15% off the original, then another 25% off what remained — 21 points of the original bill. 36% total, with the same outputs and the same eval scores.

The one we left on the table #

With cache metrics on a dashboard, something else showed up: during slow hours, the cache hit rate collapsed.

Obvious in hindsight. The default TTL is five minutes. At peak, tickets arrive constantly and every request refreshes the tier 1 entry, so it never expires. At 3am they arrive every ten or fifteen minutes: each one pays for a fresh write, gets one or two reads, and dies before the next arrives.

Bedrock offers a 1-hour TTL at a 2× write premium instead of 1.25×. A slow hour, tickets 12 minutes apart at 3 requests each:

One write covers the whole hour instead of five — roughly a 2× improvement on tier 1 off-peak.

Which sounds good until you notice it's 2× on the cheapest hours of the day. At peak the same switch would cost us money: a 2× write premium for an entry a 1.25× write would have kept alive anyway.

Doing it properly means picking the TTL from the traffic rate — a scheduler, a metric, and a new failure mode. We haven't built it, and as the client scales off-peak stops existing and the whole thing becomes pointless. That's a good reason not to build something.

I mention it because your traffic might look nothing like ours. If you run bursty, low-volume, or batch workloads with long gaps between calls, the 1-hour TTL is probably the biggest win available to you.

Do these first #

Everything above assumes the boring things are already done. We do them by default on every build, which is why caching was the first thing left to optimize rather than the first thing we reached for. If that's not true for you, start here — cheaper than caching, and they stack with it.

Consider a cheaper model. The best token optimization is a lower price per token. Route the easy paths to something small and fast — classification, routing, extraction, "does this need a human" — and save the expensive model for the turn that actually needs to reason. In a ReAct loop that's often most of your calls.

Load context lazily. Tier 2 is the interesting case. We front-load the whole player profile because we assumed the agent would want it. Does it? For plenty of tickets it never touches half. The alternative is a get_player_details

tool, or a skill that loads on demand — you trade a guaranteed cost on every ticket for an occasional extra round trip.

We haven't measured this properly yet. I suspect it beats caching tier 2 comfortably, for the same reason tier 2 caching failed: content that's expensive and rarely read shouldn't be in your prefix at all.

Conclusion #

Two things to take away.

First, caching isn't a switch you flip. It's how you lay out the prompt. Stable content first, changing content last, breakpoints in between. Add cache_control

to a prompt with a timestamp in the header and nothing is ever read back, because the prefix differs every time — no error, just a fresh entry on every call at 1.25×, which is 25% more than you paid before you turned caching on.

Second, what matters is how often a block is read, not how big it is. The instinct is to cache the biggest block; cache the most-read one instead. Often they're the same. When they aren't, like our tier 2, you get the complexity and none of the saving. Before adding a breakpoint, ask how many times that entry gets read before it expires. One or two? Skip it.

And before any of that, read your prompt. Top to bottom. Ours had 15% dead weight and we thought it was tight. Yours probably does too.

Related: How to Build ReAct Agents in 2026 · Measure What Counts: The AI Engineering Approach to Agent Evaluation