How coding agents like Cursor quietly cut input costs by reusing KV states across turns — and what actually breaks the cache

A developer discovered that Cursor, an AI coding agent, achieved nearly 99% cache read rates by reusing Key/Value states from the transformer prefill stage across turns. Prompt caching exploits prefix stability, where static content like system instructions and file context is placed first, so only new user input is recomputed, drastically reducing input costs. However, any divergence in the token prefix breaks the cache, forcing full recomputation.

I was poking around my usage dashboard in Cursor and noticed a metric I'd never paid attention to before: Cache Read . Date Type Model Tokens May 2, 06:50 PM Included auto 898K → Cache Read May 2, 06:49 PM Included auto 1.4M → Cache Write May 2, 06:47 PM Included auto 346.8K → Input / Output Almost 99% of my input tokens for that session were served from cache. That sent me down a rabbit hole on what prompt caching actually is , and why it matters so much for agentic coding tools specifically. Here's what I found. Agentic coding tools ship a lot of context on every single call — system instructions, workspace rules, file contents, conversation history, the works. LLMs are stateless by default, so naively, all of that gets reprocessed from scratch on every request. Prompt caching short-circuits this. It saves the intermediate computation from the transformer's prefill stage — specifically the Key/Value states — so that when a later request shares the same leading sequence of tokens, the model reuses those states instead of recomputing them. One important caveat: this only applies to input tokens. Output generation is still sequential, token-by-token, and gets no speedup from caching. Coding agents don't just fire your question at the model. They assemble a large composite prompt, and — critically — they structure it with the most static content first and the most volatile content last: <system instructions You are an expert coding assistant... </system instructions <project rules Use functional React components... </project rules <file context 1 ... </file context 1 <file context 2 ... </file context 2 <conversation history ... </conversation history <user input Refactor this function to handle edge cases. </user input Once that sequence hits the model, the transformer tokenizes it and starts computing attention. For every token, it derives a Key K and a Value V — essentially a map of how that token relates to every other token before it. This is the expensive part: attention computation scales quadratically with sequence length, so a big context window lots of files, long history gets disproportionately costly to prefill. The model doesn't refer back to your raw text when generating a response — it refers to these computed KV matrices. That's the artifact caching actually preserves. Providers typically charge less for input tokens than output tokens, but agentic coding tools push such enormous input payloads that shaving cost off the input side is still very much worth doing. Here's the flow, roughly: Turn 1 cold start : Turn 2...N warm : So instead of re-running the full O n² prefill on every message, only the marginal new tokens get freshly processed. That's the whole trick, and it's why long coding sessions don't get proportionally slower and more expensive turn after turn. Most major providers now do this optimization implicitly under the hood. Google's Vertex AI docs on context caching https://docs.cloud.google.com/vertex-ai/generativeai/docs/context-cache/context-cache-overview are a good primary-source reference if you want to see how one provider exposes this explicitly. Caching depends entirely on prefix stability . The moment a token diverges from what's cached, everything downstream of that point has to be recomputed. In practice, here's what kills your hit rate: Most systems rely on exact token-prefix matches, though some providers Google's docs linked above, for instance expose more explicit cache handles or partial-reuse mechanisms. This is the part that seemed contradictory at first. If attention connects every token to every other token, shouldn't adding a new message at the end of the prompt ripple backward and change how the earlier tokens system instructions, file context get attended to? It would — if the model attended in both directions. But modern generative LLMs are decoder-only and use causal masked self-attention : a token can only attend to tokens that came before it, never after. That one architectural detail is what makes caching possible at all. We're not caching a fragile, bidirectional global attention matrix — we're caching the KV tensors of the prefix . Those K and V states for the early context are computed once and never change, no matter what you append afterward. New tokens just generate their own Queries and attend against the already-cached Keys of everything before them. Prompt caching solves the "don't recompute what hasn't changed" problem. It doesn't solve the separate problem of what to put in the context window in the first place . Instead of stuffing every file into the prompt and hoping for a cache hit, coding agents increasingly rely on retrieval — RAG-style embeddings over the codebase — to selectively pull in only the relevant slices of code. That's next on my list to dig into: how coding tools build queryable views of a codebase with embeddings, and what that does to the cost equation. If you want the deeper transformer-architecture background behind KV caching and attention, 3Blue1Brown's "Large Language Models explained briefly" is a genuinely excellent, low-jargon primer. Originally posted on Susheem's Jottings — cross-posting here for the dev.to crowd.