Cache-Control for LLMs

Claude Sonnet 5 charges $2.00 per million fresh input tokens. Writing them into a five-minute cache costs $2.50, and every read after that costs $0.20 (see here). So two identical calls cost $2.70 with caching instead of $4.00. One reuse pays for the write. Every hit after that is 90% off.

For a resource priced like that, the controls are thin. Anthropic offers just two: where the reusable part of a prompt ends, marked at up to four places, one of which automatic caching may spend on its own, and how long the entry lasts, either five minutes or an hour. OpenAI offers the same four places and thirty minutes. That is the whole vocabulary. So developers improvise. The standard trick is a keepalive: replay the prefix on a timer, because every read resets the expiry clock. It works. But it also scales with how many prefixes you keep warm.

The trouble isn't a lack of cache controls. They exist, except a layer down. LMCache pins, compresses, clears, and moves KV state across GPU, CPU, and NVMe. SGLang stores and matches reusable prefixes in a radix tree. vLLM lets a caller salt the cache key. What is missing is a portable contract above all of it.

HTTP had the same gap in 1996. Caching worked, but what a server could say about it was thin: an expiry date, a conditional request, and Pragma: no-cache

. RFC 2068 fixed it by introducing additional vocabulary: Cache-Control

for policy, `ETag`

for identity, `Vary`

for what makes two requests the same. That is what LLM caching does not have.

Who knows what #

Every serving system is already solving an optimization problem: maximize hit rate within a fixed memory budget, by choosing what to keep, for how long, and who may reuse it. What the provider lacks information about what each part of the prompt is worth.

Take one request. Part of it is a system prompt that has not changed in a month and will be sent again in ten seconds. Part of it is a user turn nobody will ever send again. One is worth keeping and the other is worth dropping the moment it is read, but both arrive as the same thing: tokens, in order, with no note attached.

So the provider applies its default to both, and the user turn holds memory until eviction gets to it, taking room something reusable could have had.

Such inputs come in two kinds. Some are constraints on the solution: this content may not be retained, this reuse has to be exact, this must never cross a tenant boundary. The rest are parameters the solver can weigh or discard: this branch resumes in about thirty seconds, a miss here is expensive. Everything else, meaning placement, eviction, compression, prefetch, scheduling, and the model- and adapter-specific parts of cache identity, stays with the solver.

Supply them and the solver gets better for both sides: the same memory serves more hits, so the application pays less and the provider gets more out of the hardware it already bought.

Constraints

Constraints are non-negotiable, so no amount of memory pressure makes violating one an option. There are four important types of constraints:

Retention. Whether this state may be kept at all, and if so, for how long at most. A user's medical intake form can be cached for the length of the request and no longer.Reuse. Two settings.exact

matches only on an identical prefix: two requests share state until they diverge, and everything past that is recomputed.approximate

matches each span wherever it was cached, then repairs it. How many tokens get repaired is a choice: repair more and the result lands closer to exact but takes longer.CacheBlend, for instance, repairs 15% of tokens for roughly a 0.02 quality drop, for its benchmark. But another user may choose a different trade-off.Scope. How widely this state may be shared: request, user, tenant, a named group, or public. Since system prompts and tool schemas repeat across customers, sharing cuts cost and latency for everyone.SafeKVestimated (on three datasets) that partitioning per user raised time to first token by 2.3 to 8.9% on Llama-2-13B and 8.3 to 38.9% on Llama-2-70B.

There are safety implications to sharing. Send a candidate prefix, time the reply, and a fast one means somebody sent it before you. Because the cache is a tree, partial matches confirm partial guesses, and they compose: a2025 auditfound cross-user sharing at seven providers, and later work reconstructed whole prompts token by token. Providers now isolate by workspace or organization but the user ought to be able to specify finer or coarser boundaries.Identity. What makes two requests the same, so one can reuse the other's state. The server builds most of the key, chaining each block's hash to its parent alongside model configuration, adapters, and multimodal inputs. But that key tracks exactly one thing: the tokens sent. So identical tokens always match, even when they have stopped meaning what they meant, as when a document is re-approved under a new policy while its text stays put. Nothing in the bytes says so, and no sharing boundary helps, since it is the application's own state it needs to stop matching. A version does: label the spandocument:8841:v42

, bump it tov43

, and the miss happens on purpose.

Hints

Hints are inputs a provider can weigh or discard. Most of what an application knows about its own reuse, a provider can work out by counting hits. Two things it cannot.

The first is the future. A write costs 1.25 times fresh input and pays back only on a second read. A support bot caching a weekly corpus saves most of its input bill; a news assistant caching articles it will never see again pays more than it would with caching off. Every block is new either way, so counting cannot tell them apart until the money is spent. The same goes for a system prompt that expires at nine tomorrow, or an agent abandoning a branch: history says both are thriving right up to the moment they are worthless.

The second is worth. Two requests with identical access patterns can carry different stakes, since a miss costs a batch job some throughput and an interactive agent its latency budget. With memory to spare none of this matters, because a cache with room keeps everything. Under pressure it is the whole question, and the tiebreak needs a number only the application has.

That second one is already being specified. vLLM has an open RFC for retention priority per token range, with an optional duration and a scope, written on the explicit theory that the caller defines policy and the engine executes it.

What the provider says back #

A declarative surface without a reply is a wish list. Before the request, the application needs to know what the provider supports. After it, the application needs to know what happened: bypass, miss, exact hit, approximate hit, the scope actually applied, whether the state was retained. HTTP has a header for exactly this, Cache-Status, added years after Cache-Control because declaring policy turned out not to be enough if neither side could see what the cache did. (Vary

and a 304 do not do this job, since one selects among representations and the other only says a stored copy still stands.)

The part that needs building #

Agent harnesses avoid editing their context because prefixes nest, so a cut in the middle invalidates everything behind it. For example, removing a 6,000-token error trace from a 40,000-token context means recomputing the 18,000 good tokens that followed it. The junk rides along instead, until the window fills and the harness rebuilds the whole prompt from a summary.

Leyline removes the tradeoff. Cut the span, rotate everything downstream into its new position, and the state is valid again without recomputing a token.

There is a catch, however. Rotation fixes where those tokens sit, not what they saw. That state was computed while the deleted span was visible and still carries it, so removing the influence means recomputing from the cut after all. Deleting text and deleting its effect are not the same operation.

Which means an application asking to delete something has to say which deletion it means. Dropping a verbose log, the cheap one is fine. Retracting a false conclusion the model has already reasoned from, it is not, and the length of the span tells you nothing about which case you are in.

That makes editing the exception here. Every other control exists somewhere and reaches nobody, so the naming is the work. This one is the other way around: the name is obvious and the kernels are not finished, since Leyline's rotation is cleanest on MLA with broader attention support still open.

What exists today #

The pieces are all in the field, in different hands. Anthropic and OpenAI expose write points and fixed lifetimes. Gemini exposes named cache objects with user-set TTLs and duration billing. vLLM exposes a caller-supplied cache_salt

that enters the first block's identity, which is a namespace under another name, plus an open RFC for a salt per message. LMCache exposes lookup, movement, pinning, clearing, and compression to whoever runs the engine.

The pattern is not that providers are behind. It is that the controls thin out as you move from the operator toward the application. An operator talking to their own hardware has most of the surface. An application talking to a vendor has write points, a lifetime, and a routing key.

Provider capabilities as of August 2026, and this table will age faster than the argument:

Who should ship it first #

Provisioned throughput is the natural first market, because the customer reserves fixed processing capacity and benefits directly from skipping prefill. Azure deducts cached input tokens entirely from PTU utilization, and in Microsoft's own sizing example a 50% input cache rate takes a workload from 110 PTUs to about 80, roughly 27% fewer capacity units.

It also narrows the usual objection rather than removing it. On shared infrastructure, honoring a retention request means holding memory someone else could use, and rationing that is reasonable. On reserved capacity the capacity is already yours.

And yet PTU-managed GPT-5.6 deployments support neither explicit breakpoints nor cache write reporting, while standard paygo deployments support both. The customers with the clearest incentive have the least control over the cache producing their savings.

Postscript. See https://github.com/gojiplus/llm-cache-control for the RFC, etc.