# Your AI Agent's Bill Tripled Overnight. The Prompt Cache Broke, Not the Model. > Source: > Published: 2026-07-15 03:39:29+00:00 Nobody touched the model. Nobody touched the traffic. Nobody touched the prompts, as far as anyone could tell from the diff. And yet the API bill for our agent tripled between one deploy and the next, and average response latency to first token nearly doubled with it. The model was fine. The prompt cache was dead — and it had been killed by a single line we added to be "helpful." Most people treat prompt caching as a magic discount switch: turn it on, get cheaper calls. What it actually does is match a byte-for-byte identical prefix of your request against a cache from a previous call, up to a breakpoint you (or the SDK) declare. If the prefix matches exactly, the provider skips re-processing those tokens through the model's attention layers and charges you a fraction of the price for them — often around 10% of the input cost, with a large cut in time-to-first-token too, since the KV cache for that prefix is already computed and sitting in memory. The keyword is exact. Not "semantically similar." Not "mostly the same." One different token anywhere before the breakpoint, and the entire prefix after that point stops matching — even if 99% of it is identical to the last call. Our agent's system prompt looked roughly like this: ``` Current time: 2026-07-15T09:14:02Z User timezone: America/Chicago [~18,000 tokens of tool schemas, retrieval docs, and few-shot examples — completely static across every single call] [dynamic conversation turn] ``` Putting the current timestamp at the top felt reasonable — it's genuinely useful for the model to know what "now" means when reasoning about dates. But it meant the very first tokens of every request were unique, every time, down to the second. The cache breakpoint we'd set after the static block never mattered, because the match check fails at token one. That 18k-token block of tool definitions and docs — identical on every call — was being fully re-processed and fully billed, every single time, for months. We only caught it because someone happened to log cache-hit-rate as a metric out of curiosity. It read 0%. Not "lower than expected." Zero. Cache stability has one rule: everything static goes before the breakpoint, everything volatile goes after it. Full stop. ``` [STABLE PREFIX — cache breakpoint here] tool schemas retrieval docs few-shot examples system instructions that never change per-request [VOLATILE SUFFIX] current time session id user's live message retrieved context for this specific turn ``` If the model needs to know the current time, inject it in the volatile suffix, right next to the user's turn — not at position zero. The reordering cost us nothing in capability. It cost us months of a bill that was 3-10x higher than it needed to be, depending on the day's call volume. Do the math on your own traffic: if you're sending an 18k-token static block on 3,000 calls a day at full price versus ~10% cached price, that's the difference between paying full rate on roughly 54M tokens a day and paying full rate on effectively none of them. At any reasonable per-token rate, that gap is not a rounding error — it's most of your bill. Frameworks and agent SDKs will silently do this to you even when your own prompt template is written correctly. Two common ways: You won't see either of these by reading your prompt template, because your template is fine. You'll only catch them by diffing the actual serialized bytes sent to the API across two consecutive calls. If they're not identical up to your intended breakpoint, something upstream is injecting or reordering, and your cache hit rate is lying to you even though your code looks correct. Treat cache-hit-rate as a first-class metric, not a nice-to-have. It isn't just a performance number — for any agent making non-trivial numbers of calls with a large static context, it is your bill. What's something that quietly broke your cache without anyone noticing for weeks?