{"slug": "prompt-caching-in-agents", "title": "Prompt Caching In Agents", "summary": "Prompt caching, which reuses computed attention states from previous LLM requests, is critical for making coding agents economically viable but remains fragile to changes in tool definitions, model switches, or provider routing, according to a technical analysis. The KV cache stores key-value tensors per token and layer, enabling reuse of a matching prefix across turns, but its effectiveness depends on session affinity or explicit cache key management.", "body_md": "# Prompt Caching In Agents\n\nLarge language models are often thought of like functions: send in some text, receive some text. That is a useful abstraction, but it ignores one of the most important parts of running a coding agent: most of the input is the same as last time. In other words we mostly append to it.\n\nA coding agent sends the model its system prompt, tool definitions, project instructions, conversation history, tool calls, and tool results. On the next turn it sends almost all of that again, plus a small amount of new material. Once a session has grown to tens or hundreds of thousands of tokens, recomputing the whole prompt for every turn is slow and expensive.\n\nPrompt caching is what makes this somewhat economic, but it is also quite fragile. A changed tool definition, a model switch or a provider routing decision can turn what one would expect to be a cheap incremental request into a full replay of the context.\n\nFor coding agents, cache behavior is therefore not just an implementation detail or optimization. It affects latency, cost, tool design, session design, and even which product features should be made available.\n\n## What a KV Cache Contains\n\nA transformer processes a prompt in two broad phases. During **prefill**, it\nreads the input tokens and computes attention state for them. During **decode**,\nit produces new tokens one at a time.\n\nAt each attention layer, every processed token produces a key and a value. These\nare not quite like key-value lookups in a hash table: both are arrays of\nnumbers, usually floats or lower-precision quantized values. When processing a\nnew token, the model compares that token's **query** with the earlier **keys**\nto determine how relevant each earlier token is. It then uses those relevance\nscores to form a weighted mixture of the corresponding **values**. In that\nsense, a key is what the model matches against, while a value is the information\nit retrieves (but the lookup is fuzzy rather than \"returning a single exact\nmatch\" like a dictionary lookup.)\n\nThose keys and values are retained so that the next generated token can attend\nto everything that came before without recomputing the earlier tokens. This\nretained state is the **KV cache**.\n\nConceptually, a request looks like this:\n\n```\nrequest 1:\n\n[system][tools][user][assistant][tool result][user]\n<--------------------- prefill -------------------->\n                       |\n                       K and V tensors per token and layer\n\nrequest 2:\n\n[system][tools][user][assistant][tool result][user][new]\n<---------------- reusable prefix ----------------><--->\n                                                    |\n                                                    new work\n```\n\nThe real representations are more complicated, model-specific, and \"quite\" large. The important property is that they correspond to a particular token prefix. Two prompts that mean the same thing but tokenize differently do not share a KV cache. If a token changes in the middle, everything after that token is a different continuation.\n\nPrompt caching extends the lifetime of this state beyond one generation. When the next API request from the coding agent begins with the same tokens, the inference system can reuse the stored work for the matching prefix and prefill only the new suffix. So far, the theory.\n\n## Where the Cache Lives\n\nIn order for a cache to work it needs to be stored somewhere, and it needs to be addressable. There are two broad ways inference systems make KV caches available to a later request.\n\nThe simpler approach is **session affinity**. It works by keeping the KV cache\non or near the GPU that computed it, and routing the next request back to the same\nworker. A session ID or prompt-cache key becomes a trivial routing hint and so\nyou can potentially even deal with this problem on the HTTP load balancer level\nwithout having to look into the payload.\n\n``` php\nrequest(session-42) --> router --> worker 7 --> GPU 7 KV cache\nnext(session-42)    --> router --> worker 7 --> GPU 7 KV cache\n```\n\nThis avoids moving a very large cache over the network. It is fast when it works, but it constrains scheduling. The selected worker can become overloaded, restart, or evict the entry. A router may also decide that balancing the fleet is more important than preserving one session's cache. It is however a very attractive solution because it works with little extra deployed infrastructure and hardware.\n\nThe other approach is to **distribute the cache**. KV blocks can be stored in\nanother memory tier or made available across workers, so a request is not tied\nas tightly to one GPU.\n\n``` php\n                         +--------------------+\nrequest --> scheduler -->| worker 3 / GPU 3   |\n             |           +--------------------+\n             |\n             +----------> distributed KV blocks\n             |\n             +----------> worker 9 / GPU 9\n```\n\nThat improves scheduling flexibility and recovery, but moving, indexing, and retaining KV blocks is itself a systems problem. Implementations mix GPU memory, host memory, local storage, remote storage, prefix-aware routing, and eviction policies in different ways.\n\nTo put KV caches into perspective: they can be large but they are in some ways smaller than one would assume. With various tricks, the size of KV caches can be reduced to a handful of gigabytes, even for long conversations.\n\n## Caches and Prefixes\n\nPi sessions are trees, not lists. `/tree`\n\ncan move the active conversation back\nto an earlier point and continue along another branch. A rewind can discard the\nactive suffix without deleting it from the session file. A new branch can share\nmost of the old context, a little of it, or effectively none of it. This design\nis not unique to Pi, quite a few coding agents have something at least\nconceptually similar. Even if you do not represent the session as a tree,\nit's not uncommon for agents to have some form of rewinding.\n\n```\n                             +-- E -- F  another branch\n                             |\nsession S: root -- A -- B -- C -- D  current branch\n                   |\n                   +-- Z  branch near the start\n```\n\nAll three branches can have the same Pi session ID. From the router's perspective they are one session. From the prompt cache's perspective they are three token sequences with only partial prefix overlap.\n\nIf the cache keeps reusable prefix blocks, jumping from `D`\n\nto `F`\n\nmay still\nreuse `root -> C`\n\n. If it only retains the hottest continuation, if the shared\nblocks were evicted, or if the request is routed elsewhere, the hit can be much\nsmaller. Jumping to `Z`\n\nmay preserve only the system prompt and initial tool\ndefinitions even though it starts from `A`\n\n. The precise cache management\nbehavior here depends greatly on the providers.\n\nThe reverse can also happen. `/fork`\n\nor a new session can produce a new session\nID while carrying over a large amount of identical context. A routing system\nthat isolates caches by session key may fail to notice that useful overlap.\n\nThe reusable prefix determines what work can be cached. Session identity merely helps the infrastructure find likely content. On some systems the routing key is crucial to manage caches, on others it's merely an optimization.\n\n## Explicit vs Automatic Prefix Caching\n\nProvider APIs expose caching in two main styles.\n\nAnthropic's traditional interface uses explicit `cache_control`\n\npoints. The\nclient marks boundaries after stable parts of the request, such as the system\nprompt, tool definitions, or the latest cacheable conversation content. The\nserver can then write or look up the prefix ending at those points. The boundary\nis explicit, but reuse still requires the content before it to match. Not only\nare the cache points explicit, so is the pricing. You pay for cache writes, and\nyou get to choose for how long which comes at different price points.\n\nOther APIs use automatic prefix caching. The client sends the request normally, and the provider finds a reusable prefix without client-placed breakpoints. A prompt-cache key or session header may improve routing or grouping, but it does not make different prefixes equal.\n\n## Why Tool Loadouts Trash Caches\n\nTool definitions usually appear before the conversation and they are \"folded\" into the system prompt internally. Their names, descriptions, and JSON schemas are model input just like any other text. Adding one tool, removing one, changing its schema, or even serializing the tools in a different order can move the first mismatch close to the start of the prompt.\n\n```\nturn 1: [system][read][write][bash][conversation...........]\nturn 2: [system][read][write][bash][deploy][conversation...]\n                                   |\n                                   old conversation is now\n                                   after a mismatch\n```\n\nThis is a common surprise with plugin systems and MCP-style tool catalogs. Loading a tool only when it becomes relevant sounds efficient because fewer schemas are sent initially. On most models, however, the newly expanded loadout invalidates the cached conversation that follows it. Saving a few tool-schema tokens can cause tens of thousands of conversation tokens to be processed again.\n\nSome newer model APIs support **additive tool loading**. A tool can become\navailable at a specific tool result inside the transcript instead of being\ninserted into the original tool list. The old prefix remains unchanged:\n\n```\n[system][initial tools][conversation][new tool][next turn]\n<--------- cached prefix ----------->\n```\n\nPi nowadays supports this for models with native deferred-tool mechanisms. When\nan extension makes a purely additive change with `setActiveTools()`\n\n, Pi records\nthe added names on the tool result. For supported Anthropic models it uses\ndeferred definitions and a `tool_reference`\n\nand for supported OpenAI models it\nemits the corresponding tool-search items. Other models get a safe fallback: Pi\nsends the complete active tool list on the next request, which works\nfunctionally but may wipe the prompt cache.\n\nThe word **additive** matters as removing tools, replacing one loadout with\nanother, or changing prompt snippets still changes earlier input. An extension\nthat rebuilds the system prompt, shuffles tool order, injects timestamps, or\nchanges active tools every turn can accidentally defeat caching for the entire\nsession.\n\nExtensibility means Pi cannot guarantee cache stability on behalf of every extension. We can provide cache-friendly mechanisms; extensions still have to use them and from what we have seen, for many extensions cache efficiency is an afterthought. This is, in part, because when you pay on a fixed subscription the associated cost with cache misses is not quite as obvious.\n\n## Interruptions and TTLs\n\nSome important prompt caches have short default lifetimes. Anthropic's default five-minute cache is particularly important because it is shorter than many normal coding activities. If you go to sip a coffee when using Fable, and you come back 10 minutes later, a single \"say hi\" message will cost you more money than you expect.\n\nThat's because while the user may think of a coding session as continuously active, the inference provider sees a sequence of isolated requests:\n\n``` php\nmodel request --> run tests for 7 minutes --> model request\n                  no cache traffic here\n```\n\nA long build, a test suite, lunch, a meeting, or simply stopping to review a diff can outlive the cache. The next request contains the same prompt, but the stored KV state is gone and the prefix is billed again as input.\n\nSince Pi is currently not a permitted harness on Anthropic's subscription we're following the 5 minute default that Anthropic recommends for API users. However from looking at Claude Code's codebase we know that for their own subscription users, they are increasing that cache timeout to one hour. The increased cost of this however often is not worth it, when you need to pay API token prices.\n\nBut you can opt into this. Some providers such as Anthropic expose longer\nretention controls. For supported direct APIs, Pi users can set\n`PI_CACHE_RETENTION=long`\n\nto request them. That is still only a request: Pi\ncannot force a gateway to retain an entry, prevent eviction under memory\npressure, or keep a cache alive while no model request is being made.\n\n## The Price of a Miss\n\nProviders usually price uncached input, cache writes, and cache reads differently. Cache reads are commonly discounted because the expensive prefill work has already happened. Cache writes can carry a premium because the provider is promising to retain state for later use.\n\nImagine a coding session with 100,000 tokens of history followed by a short new request as the Fable example above. When the cache works, almost all of that history is charged at the lower cache-read price. Only the small amount of new material needs to be processed at the regular input price and potentially written to the cache.\n\nWhen the cache misses, the provider has to process the entire 100,000-token\nhistory again at the regular input price. It may also charge to write that\nhistory back into the cache. This is why a short request such as `continue`\n\ncan\nbe surprisingly expensive after a cache expires. In a long coding session,\nre-reading the old input can cost much more than generating the next answer.\n\nCaching also has a chance to create non-obvious incentives.\n\nThe user should want high hit rates because they reduce latency and price. An inference operator that owns the GPUs should want them too: less prefill work means more requests served with the same hardware. A well-designed cached-token discount can align both sides while leaving the operator with better margins.\n\nA gateway or reseller can have a different incentive. If it earns revenue from input tokens billed at the uncached rate, a cache miss can produce a larger customer invoice than a hit. Whether that also produces more profit depends on its upstream costs, contracts, and who operates the cache. In a badly aligned stack, the party responsible for routing may not bear the full cost of misses, while the party billing the user earns more revenue when they happen.\n\nThat does not mean providers sabotage caches but it means cache performance should be observable. Users should not have to infer that only from a surprisingly large bill. Understanding if something odd is going on with caches can be an important insight.\n\nStrict cache adherence also means less flexibility for a gateway to route you to the best option in-between turns. You might want to take a cache hit to continue with a different model which from that point onwards might be more economical, or it might be the case that you might be better off load balancing to another provider.\n\n## Why Pi Does Not Prune Aggressively\n\nNow that you've made it this far, you probably have an idea why Pi does not prune tool calls. It is tempting to control cost by continuously deleting old tool results or rewriting history and sometimes that is necessary, especially near the context-window limit. But as we have learned, pruning has a cache cost of its own.\n\nDeleting content from the middle changes the prefix at the deletion point. All surviving conversation after it may need to be processed again. The immediate cost of rewriting a long cached context can exceed the future savings from removing a small number of cheaply cached tokens.\n\nA rough break-even comparison is:\n\n```\none-time rewrite cost\n    ~= surviving tokens after the edit * (uncached price - cache-read price)\n\nfuture savings per turn\n    ~= pruned tokens * cache-read price\n```\n\nThis is not only an accounting question as old tool results often contain the evidence the model used to make later decisions. Removing them can degrade behavior even if a summary preserves the gist.\n\nPi therefore prefers a stable, append-oriented transcript and does not treat every old token as waste. Compaction is available when context pressure justifies a lossy rewrite. Because compaction deliberately creates new context rather than accidentally re-billing an unchanged prompt, Pi treats it as a cache reset rather than a cache failure in its session statistics.\n\nThe goal is not the smallest possible prompt but the best trade-off among model context, cache reuse, latency, and price.\n\nSimultaneously there can be a case for pruning too. If you are working with providers that do not discount you for good cache usage, or it's for whatever reason not possible to get high cache rates, it might be preferable to prune. It definitely improves the opportunity for the router to balance between different backends as caches are not transferable.\n\n## What Pi Can and Cannot Do\n\nPi works to keep stable inputs stable. It passes a consistent session ID and provider-specific cache hints, places explicit cache points for APIs that require them, records cache-read and cache-write usage, and supports message-anchored additive tool loading where models allow it. Its default transcript behavior also avoids gratuitously rewriting old context.\n\nPi cannot control every layer after the request leaves the machine. It cannot choose a provider's eviction policy, extend a cache beyond what the API permits, keep a particular GPU alive, or guarantee that a gateway honors affinity. It also cannot preserve a prefix that an extension changes.\n\nWhat it can do is make cache health visible.\n\nThe interactive footer shows cumulative cache reads and writes as `R`\n\nand `W`\n\n,\nplus `CH`\n\nfor the latest request's cache-hit rate. The `/session`\n\ncommand gives\na fuller view: total cached and uncached input, cumulative hit rate, cost, and\nan estimate of tokens and dollars re-billed by [significant cache misses](https://github.com/earendil-works/pi/blob/34f3719a942ecbf3e6d23e67098f47ba2867de0a/packages/coding-agent/src/core/cache-stats.ts#L50-L90).\n\n```\nMessages\nTotal: 178\nUser: 6\nAssistant: 58\nTools: 114 calls, 114 results\n\nTokens\nInput: 7,129,883\n  Cached: 6,776,832 (95.0%)\n  Uncached: 353,051\nOutput: 30,013\nTotal: 7,159,896\n\nCost\nTotal: $6.054\nCache Re-billed: $0.728 (161,744 tokens, 2 misses)\n```\n\nUsers who want misses called out as they happen can enable **Show cache miss\nnotices** in `/settings`\n\n, corresponding to `showCacheMissNotices`\n\nin\n`settings.json`\n\n. Pi then inserts a warning after a significant miss, including\nthe estimated re-billed tokens and cost. When it can observe a model switch or\nan idle gap beyond the usual short TTL, it says so. For other misses it reports\nthe fact without pretending to know what happened inside the provider.\n\n## Common Reasons for Worse Cache Performance\n\nWhen a session's cache-hit rate looks wrong, the usual causes are:\n\n**Idling.** A command, review, or conversation pause exceeds the provider's retention window.**Model or provider switches.** KV state is model-specific and generally does not move across providers.**Branch navigation.**`/tree`\n\n, rewinds, forks, and alternate branches can change the active token sequence even when the session ID remains the same.**Compaction or manual history rewriting.** These intentionally replace part of the prompt and establish a new prefix.**Tool and reasoning level changes.** Adding, removing, reordering, or editing tool definitions changes an early part of the request unless the model supports message-anchored loading and the change is purely additive. Reasoning level changes usually have the same effect.**Dynamic system prompts.** Timestamps, random values, changing project context, and extension-provided prompt snippets can invalidate everything after them.**Extension context transforms.** An extension that modifies old messages or provider payloads can make an apparently stable Pi transcript unstable on the wire.**Provider routing and eviction.** The prompt can be identical and still miss because the relevant KV blocks are no longer available where the request lands.", "url": "https://wpnews.pro/news/prompt-caching-in-agents", "canonical_source": "https://earendil.com/posts/prompt-caching/", "published_at": "2026-07-23 18:04:14+00:00", "updated_at": "2026-07-23 19:27:10.298690+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-infrastructure", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/prompt-caching-in-agents", "markdown": "https://wpnews.pro/news/prompt-caching-in-agents.md", "text": "https://wpnews.pro/news/prompt-caching-in-agents.txt", "jsonld": "https://wpnews.pro/news/prompt-caching-in-agents.jsonld"}}