I spent weeks chasing the wrong theory about why my agent was expensive.
The theory was the obvious one: expensive context means large context. So I built what seemed like the right fix — a routine that re-scored the tool set on every turn and sent only the tools relevant to that particular message. Fewer tokens out, smaller bill.
Here is what it actually did, measured in production on three trivial turns of the same conversation:
| before | after | |
|---|---|---|
| tools, turns 1→3 | 52 → 54 → 61 | 59 · 59 · 59 |
| cache hit rate | 9.6% | 80–88% |
| cost per turn | $0.0168 | $0.0036–0.0046 |
Same model. Same conversation. A 75% reduction on the cached turn, and I got it by removing the optimization.
Provider prompt caching matches the prefix byte for byte, up to the first difference. Your tool schema is part of that prefix.
My routine re-scored tools by embedding the user's message. Different message, different relevance ranking, different tool set: 52, then 54, then 61 tools in a single conversation. Three turns, three prefixes, three full-price cache writes.
I was paying full input price on every turn in order to save the handful of tokens I had trimmed.
Pruning is worth about 4% of what the cache is worth. And in my case, pruning was the thing destroying the cache. The optimization was negative-value by more than an order of magnitude, and it looked like good engineering the whole time.
The tool-count ceiling was sized by a learned factor — a running average calibrated from real usage. That same factor also fed a budget guard.
Those two consumers want opposite things. The budget guard wants accuracy: it should track reality as closely as possible. The tool ceiling wants stability: it should not move, because every move rewrites the prefix. One number cannot serve both. Worse, the learned factor was not persisted, so every deploy restarted the drift from scratch.
The fix was to split it: a fixed factor sizes the ceiling, the learned factor stays in the budget guard only, and trivial turns use a stable tool set with no per-message embedding.
Every guide to prompt caching lists the same culprits: a timestamp in the system prompt, user-specific content in the cached prefix, inconsistent whitespace, dumping raw files into context. All of those are carelessness. You read the list, you check your prompt, you move on.
Mine passed every item on that list. The thing invalidating my prefix was a deliberate, well-engineered optimization whose entire purpose was to reduce cost. That is a different failure class, and it is harder to find, because you are not looking for a bug inside the code you wrote to fix the problem.
There is prior work on the general shape of this. Don't Break the Cache (arXiv 2601.06007, Lumer et al.) evaluated caching strategies across 500 agent sessions with 10,000-token system prompts and found 41–80% cost reduction from controlling the cache boundary, plus the counterintuitive result that naive full-context caching can increase latency. Their framing is about where to put the boundary. Mine is one layer up: what in your own system is moving the boundary without telling you.
Before asking how much context you send, ask what changes between turns. The system prompt, the tool schema and the tool order are a stable contract, not space to squeeze. Cost work starts by measuring cache hit rate, not size.
And a corollary that generalizes past caching: when one value serves two consumers with opposite needs, it is two values. Ask each consumer whether it wants accuracy or stability. If they disagree, split it.
This regression produces no error, never turns a test red, and is invisible in the UI. It shows up on the invoice, weeks later. So the check that catches it is not a behavioral test — it is a test that asserts the ceiling is a constant, and fails if anyone makes it learned again.
These numbers come from a multi-tenant agent system I have been building solo since June, logged in a dated engineering diary. Every figure above is reproducible from a command recorded alongside it.