{"slug": "kv-cache-rules-everything-around-me", "title": "(KV) Cache Rules Everything Around Me", "summary": "Cache reads, not output tokens, dominate the cost of running AI agents, according to an analysis of Nvidia's Open-SWE-Traces dataset priced with Fable 5 that found output tokens account for just 9-18% of the total bill. The same workload costs $1.97M at Claude Code's reported 89% cache hit rate versus $1.07M at a 100% hit rate, an 84% increase, and improving the hit rate from 95% to 98% saves roughly 17%. The analysis attributes the pattern to agent loops re-reading the full past context on every tool call, making session cost grow quadratically with length and tool-call count.", "body_md": "### Background\n\nPop quiz: what’s the most important cost when running an agent?\n\n*Total amount billed broken down by %. Assuming Fable 5 + 100% cache hit rate. Only the additional cache write cost is separated from input cost.*\n\nIf you’re like most AI engineers, your first two guesses are wrong. Output tokens? Per token outputs are the expensive ones, but they end up being a small slice of your final bill. Input tokens? Kinda, but not the ones you write in your prompts + messages. The answer is cache reads costs developers the most (or cache writes, if your cache hit rate isn’t great). Technically both are input tokens, just at very different prices.\n\nAn agent is a loop: the model reads the context then emits a tool call, the tool returns some tokens, and the whole thing goes back into the model. Every turn re-reads everything that came before it. With a context of N tokens and T tool calls, you process roughly N × T total input tokens (cached and not) to produce a comparatively tiny number of output tokens. And since N grows with every turn, the cost of a session grows quadratically with its length *and even more with increasing tool calls*.\n\n### The cost of an agent\n\n#### Non-infra-engineer summary\n\n- When generating an output token, the entire past context (KV cache) needs to be read.\n- You aren’t charged for the KV cache, because… it is already cached.\n- When an agent stops to call a tool then resumes, the entire past context needs to be read again.\n- You are charged for the KV cache this time, because… fuck you.\n- Also that’s the largest part of your bill now because agents have a LOT of tool calls.\n\n#### Cache Cash\n\nWhen Martin Alderson [modeled a 100-turn session](https://martinalderson.com/posts/watch-out-for-cache-read-costs/) from 60k tokens of context, cache reads made up ~76% of his Opus 5 bill. I wanted real-world trajectories, so I wrote a small simulator that replays [Nvidia’s Open-SWE-Traces](https://huggingface.co/datasets/nvidia/Open-SWE-Traces) (tokens counted with the gpt-oss tokenizer, cache writes at 1.25x input price) with Fable 5 pricing:\n\nOutput tokens, which most people associate with the big price tag, are just 9-18% of the total bill. The rest is the model re-reading its own history.\n\nHere is the same dataset priced across various models, assuming a 100% cache hit rate:\n\nThose are the costs with perfect caching. At Claude Code’s reported 89% hit rate, the same workload costs $1.97M instead of $1.07M: 84% more!!\n\nSo small changes in cache hit rate can make a big difference to the total bill. Here are the rates [measured by the OpenCode team](https://x.com/thdxr/status/2085560180045975626):\n\nNote that [others have measured Claude Code at 95%](https://www.claudecodecamp.com/p/i-tried-to-reverse-engineer-claude-code-s-usage-limits), so take the exact rates with a grain of salt. But even going from 95% to 98% saves roughly 17% of the bill on this workload.\n\n### Why cache reads are (almost) free (but not for devs)\n\n#### Ingredient 1: Input tokens - prefill costs compute.\n\nPrefill is the step where the model reads the prompt. All of the input tokens go through the forward pass at once, which is very efficient on a GPU, and the cost is proportional to the number of tokens. This is what you’re paying for with “input tokens”. It’s also the only step a cache can save: a cache write is a prefill where the provider keeps a preprocessed representation of the input tokens (“the activations”): this is what the KV cache is.\n\nCache reads are input tokens that do not need compute because the results were saved.\n\n#### Ingredient 2: Output tokens - decode costs memory bandwidth.\n\nDecode is the step where the model generates tokens, one at a time. To generate each token, the model has to attend over the entire KV cache, which means reading the *whole* context out of high bandwidth memory (HBM) for every single output token. There is very little compute involved but an enormous amount of memory traffic. This is why output tokens cost 5x more than input tokens. Decode is the real bottleneck of inference.\n\nEvery single output token needs to read the entirety of the input, but luckily they are already cached.\n\n#### Ingredient 3: A tool call is just a pause.\n\nWhen an agent stops to run a tool, the KV cache for the whole context is kept in memory. Nothing has to be recomputed. The only cost is keeping those bytes around while the tool runs, and keeping bytes around uses neither compute nor HBM bandwidth (the two things inference is actually constrained by).\n\nSo my claim is that the cost of cache reads is primarily already included in the price of output tokens, at the decoding step.\n\nThis is consistent with provider actions:\n\n- Cache reads don’t count towards [input token rate limits on the Claude API](https://platform.claude.com/docs/en/api/rate-limits) . Rate limits exist to protect capacity, so that’s Anthropic telling you what a cache read costs them.\n\n- OpenAI ran automatic prefix caching for over a year without charging anything extra for cache writes; you can’t serve LLMs at scale without proper caching The 1.25x cache write premium [only arrived with GPT-5.6](https://openai.com/index/previewing-gpt-5-6-sol/) , to match Anthropic.\n\n### How is inference so profitable all of a sudden?\n\nFrontier models are more of a commodity than ever, open-weight models are on the rise, and yet the labs’ reported inference margins keep climbing. How?\n\n*Share of tokens to open-weight models from [Vercel’s AI gateway](https://vercel.com/blog/deepseek-overtakes-google-on-volume-cost-per-token-falls)*\n\nThe answer of course is the KV cache (+ the agentic change of workload): more tool calls means more cache reads, which means more profit.\n\nIt also explains the “subsidized” subscription plans. [SemiAnalysis found](https://x.com/SemiAnalysis_/status/2091631658973671900) that the $200/month Claude plan can yield up to $8,000/month of API-equivalent tokens, and OpenAI’s up to $14,000/month. At list price that’s a 40-70x subsidy, which sounds insane, until you remember that 90%+ of a coding agent’s tokens are cache reads and the true cost of serving them is a small fraction of the sticker price.\n\nWhich is also why, if you run agents at scale, self-hosting makes way more sense than it does for chat. Self-hosting an open weight model is the only way to keep the cheapness of the KV cache for yourself rather than handing it to the lab. At least until cache read prices fall further. Fable 5.1 cut them from $1.00 to $0.25 per million tokens, which suggests they will keep dropping.\n\n### Why routing doesn’t work (for agents)\n\nThe pitch for routing is: use a cheap model for the easy steps and the expensive model for the hard ones, and save on output tokens. But output is the small slice of the bill. The moment a second model touches the context, you end up paying for it from scratch, since a KV cache isn’t shared between models.\n\nAssumptions: 100k tokens in an Opus 5 session, next step generates 1k tokens and gets 3k back from a tool. Compare Opus[1] doing it all or routing to Sonnet then back.\n\nThe tiny amount saved on output tokens was dominated by cache costs, so the “easy” step cost almost 3x more, and it was done by a dumber model. 🙃\n\nIf a router picks a model per request, it pays for this cache thrash every time, especially on the longest priciest contexts. The only way to amortize the handoff is to leave the cheap model in charge for a while, which is what subagents do, but…\n\nSubagents mostly don’t work either (if you want them to share the full context[2]). Because output tokens are a minority of the cost and you end up paying to write the whole context into both models’ cache, it’s rarely worth it.\n\n### Conclusion\n\nThe economics of agents come down to one question: how many times does the context get read, and who pays what for each read.\n\nIf you build agents, measure your cache hit rate and the number of tool calls. The number of tool calls matters way more than the number of tokens per call. And if you run agents at scale, consider self-hosting to keep those cache savings for yourself.\n\nAnd if breaking free from the tyranny of the KV cache sounds interesting to you (if only there was a new kind of foundation model that could help 🤫), reach out! Would love to jam on designs, ideas, and sci-fi.\n\n*Big thanks to Ke Deng, Kevin Zhang, and Sasha Sheng for helping me write/review this post.*\n\n[1] AFAICT APIs don’t currently have a way to say “also cache my output” - this is *perhaps* a blindspot for agentic workloads\n\n[2] The “mostly” is because subagents work great when they *don’t* need the parent’s context. If the parent says “search the codebase for X and report back in 200 tokens”, the brief is tiny, and more importantly the 50k tokens of grep output never enter the parent’s context, which makes N smaller for every turn after.", "url": "https://wpnews.pro/news/kv-cache-rules-everything-around-me", "canonical_source": "https://www.completeskeptic.com/p/kv-cache-rules-everything-around", "published_at": "2026-09-25 00:08:03+00:00", "updated_at": "2026-09-25 00:29:48.542397+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-agents", "artificial-intelligence"], "entities": ["Nvidia", "Open-SWE-Traces", "Claude Code", "Martin Alderson", "OpenCode", "Fable 5", "Opus 5"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/kv-cache-rules-everything-around-me", "markdown": "https://wpnews.pro/news/kv-cache-rules-everything-around-me.md", "text": "https://wpnews.pro/news/kv-cache-rules-everything-around-me.txt", "jsonld": "https://wpnews.pro/news/kv-cache-rules-everything-around-me.jsonld"}}