{"slug": "how-coding-agents-like-cursor-quietly-cut-input-costs-by-reusing-kv-states-turns", "title": "How coding agents like Cursor quietly cut input costs by reusing KV states across turns — and what actually breaks the cache", "summary": "A developer discovered that Cursor, an AI coding agent, achieved nearly 99% cache read rates by reusing Key/Value states from the transformer prefill stage across turns. Prompt caching exploits prefix stability, where static content like system instructions and file context is placed first, so only new user input is recomputed, drastically reducing input costs. However, any divergence in the token prefix breaks the cache, forcing full recomputation.", "body_md": "I was poking around my usage dashboard in Cursor and noticed a metric I'd never paid attention to before: **Cache Read**.\n\n```\nDate            Type       Model    Tokens\nMay 2, 06:50 PM  Included   auto     898K   → Cache Read\nMay 2, 06:49 PM  Included   auto     1.4M   → Cache Write\nMay 2, 06:47 PM  Included   auto     346.8K → Input / Output\n```\n\nAlmost 99% of my input tokens for that session were served from cache. That sent me down a rabbit hole on what prompt caching actually *is*, and why it matters so much for agentic coding tools specifically. Here's what I found.\n\nAgentic coding tools ship a *lot* of context on every single call — system instructions, workspace rules, file contents, conversation history, the works. LLMs are stateless by default, so naively, all of that gets reprocessed from scratch on every request.\n\nPrompt caching short-circuits this. It saves the intermediate computation from the transformer's prefill stage — specifically the Key/Value states — so that when a later request shares the same leading sequence of tokens, the model reuses those states instead of recomputing them.\n\nOne important caveat: **this only applies to input tokens.** Output generation is still sequential, token-by-token, and gets no speedup from caching.\n\nCoding agents don't just fire your question at the model. They assemble a large composite prompt, and — critically — they structure it with the most static content first and the most volatile content last:\n\n```\n<system_instructions> You are an expert coding assistant... </system_instructions>\n<project_rules> Use functional React components... </project_rules>\n<file_context_1> ... </file_context_1>\n<file_context_2> ... </file_context_2>\n<conversation_history> ... </conversation_history>\n<user_input> Refactor this function to handle edge cases. </user_input>\n```\n\nOnce that sequence hits the model, the transformer tokenizes it and starts computing attention. For every token, it derives a Key (K) and a Value (V) — essentially a map of how that token relates to every other token before it. This is the expensive part: attention computation scales quadratically with sequence length, so a big context window (lots of files, long history) gets disproportionately costly to prefill.\n\nThe model doesn't refer back to your raw text when generating a response — it refers to these computed KV matrices. That's the artifact caching actually preserves.\n\nProviders typically charge less for input tokens than output tokens, but agentic coding tools push such enormous input payloads that shaving cost off the input side is still very much worth doing.\n\nHere's the flow, roughly:\n\n**Turn 1 (cold start):**\n\n**Turn 2...N (warm):**\n\nSo instead of re-running the full O(n²) prefill on every message, only the marginal new tokens get freshly processed. That's the whole trick, and it's why long coding sessions don't get proportionally slower and more expensive turn after turn.\n\nMost major providers now do this optimization implicitly under the hood. Google's Vertex AI docs on [context caching](https://docs.cloud.google.com/vertex-ai/generativeai/docs/context-cache/context-cache-overview) are a good primary-source reference if you want to see how one provider exposes this explicitly.\n\nCaching depends entirely on **prefix stability**. The moment a token diverges from what's cached, everything downstream of that point has to be recomputed. In practice, here's what kills your hit rate:\n\nMost systems rely on exact token-prefix matches, though some providers (Google's docs linked above, for instance) expose more explicit cache handles or partial-reuse mechanisms.\n\nThis is the part that seemed contradictory at first. If attention connects every token to every other token, shouldn't adding a new message at the *end* of the prompt ripple backward and change how the *earlier* tokens (system instructions, file context) get attended to?\n\nIt would — if the model attended in both directions. But modern generative LLMs are **decoder-only** and use **causal (masked) self-attention**: a token can only attend to tokens that came *before* it, never after.\n\nThat one architectural detail is what makes caching possible at all. We're not caching a fragile, bidirectional global attention matrix — we're caching the **KV tensors of the prefix**. Those K and V states for the early context are computed once and never change, no matter what you append afterward. New tokens just generate their own Queries and attend against the already-cached Keys of everything before them.\n\nPrompt caching solves the \"don't recompute what hasn't changed\" problem. It doesn't solve the separate problem of *what to put in the context window in the first place*. Instead of stuffing every file into the prompt and hoping for a cache hit, coding agents increasingly rely on retrieval — RAG-style embeddings over the codebase — to selectively pull in only the relevant slices of code.\n\nThat's next on my list to dig into: how coding tools build queryable views of a codebase with embeddings, and what that does to the cost equation.\n\n*If you want the deeper transformer-architecture background behind KV caching and attention, 3Blue1Brown's \"Large Language Models explained briefly\" is a genuinely excellent, low-jargon primer.*\n\n*Originally posted on Susheem's Jottings — cross-posting here for the dev.to crowd.*", "url": "https://wpnews.pro/news/how-coding-agents-like-cursor-quietly-cut-input-costs-by-reusing-kv-states-turns", "canonical_source": "https://dev.to/susheem-k/how-coding-agents-like-cursor-quietly-cut-input-costs-by-reusing-kv-states-across-turns-and-what-49fe", "published_at": "2026-07-30 19:11:26+00:00", "updated_at": "2026-07-30 19:36:14.176575+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-infrastructure", "developer-tools"], "entities": ["Cursor", "Google", "Vertex AI"], "alternates": {"html": "https://wpnews.pro/news/how-coding-agents-like-cursor-quietly-cut-input-costs-by-reusing-kv-states-turns", "markdown": "https://wpnews.pro/news/how-coding-agents-like-cursor-quietly-cut-input-costs-by-reusing-kv-states-turns.md", "text": "https://wpnews.pro/news/how-coding-agents-like-cursor-quietly-cut-input-costs-by-reusing-kv-states-turns.txt", "jsonld": "https://wpnews.pro/news/how-coding-agents-like-cursor-quietly-cut-input-costs-by-reusing-kv-states-turns.jsonld"}}