{"slug": "the-kv-cache-is-a-market-not-a-cache", "title": "The KV Cache Is a Market, Not a Cache", "summary": "A developer argues that the KV cache in inference servers should be treated as an admission-control and pricing problem, not a memory-management one. They contend that eviction policies like LRU fail because KV blocks have wildly different future value depending on workload shape (agentic loops, RAG, multi-turn chat vs single-shot), and that treating all blocks equally leads to latency spikes and throughput loss.", "body_md": "TL;DR —Most inference servers treat the KV cache as a simple LRU or capacity buffer, but that model ignores the real problem: KV blocks have wildly different future value depending on workload shape (agentic loops, RAG, multi-turn chat vs single-shot). Eviction policy is actually an admission-control and pricing problem, not a memory-management one. Teams that keep tuning batch size and quantization while treating the cache as dumb storage are leaving throughput and latency on the table.\n\nEvery serving stack treats the KV cache the same way: as a pool of memory that fills up, gets evicted under pressure, and gets refilled. LRU, or something close to it, decides what dies. This is the wrong mental model. A KV cache entry isn't a cached value with a stable reuse probability, like a web asset or a database page. Its value depends entirely on what kind of request produced it — and that's information your eviction policy almost never uses.\n\nClassic caching theory assumes locality: things accessed recently or frequently are likely to be accessed again soon. That assumption holds reasonably well for a single-shot Q&A workload. It falls apart the moment you're serving agents, multi-turn chat, or RAG pipelines with long-lived contexts.\n\nAn agent loop might go quiet for seconds while a tool call executes, then come back needing its entire prefix reinstated instantly. A multi-turn chat session might sit idle for minutes between user messages but represents a near-certain future hit. A one-off RAG query, by contrast, produces a KV block that will almost never be touched again once the response streams out. All three look identical to a generic eviction policy: they're just memory ranges with a last-access timestamp. But their expected future value is completely different, and that value is knowable — it's a property of the workload, not the token stream.\n\nTreating all three the same way means you're either evicting high-value agent state to make room for RAG garbage, or holding onto RAG garbage \"just in case\" while agent sessions stall waiting for memory. Either way, you're paying a latency tax that has nothing to do with your model, your batch size, or your quantization scheme — it's a scheduling mistake wearing a memory-management costume.\n\nThis gets sharper once you add continuous batching into the picture. A continuous batching scheduler decides, on every iteration, which requests get to advance a token and which wait. That decision is gated by available KV memory. So your eviction policy isn't just choosing what to delete — it's implicitly choosing which requests are allowed to exist right now.\n\nIf your policy evicts the wrong blocks, you don't just lose a cache hit — you force a full prefill recomputation for a request that should have been a near-free decode continuation. That recomputation competes for the same GPU cycles as everyone else's active decode steps, which is exactly the kind of prefill-vs-decode interference that tanks p99 latency in busy clusters. The eviction decision made ten seconds ago is now showing up as a latency spike in someone else's request. Nobody will trace that back to the cache policy, because the cache policy doesn't look like a scheduler. It looks like a memory allocator. That disguise is the whole problem.\n\nShared prefix caching — reusing KV state for a common system prompt or retrieved document across many concurrent requests — is one of the highest-leverage optimizations available right now. It's also the point where the \"cache\" framing definitively stops working, because now you have contention between tenants for the same physical memory, and no mechanism to express that some prefixes are worth protecting more than others.\n\nA shared prefix used by every request in a high-traffic agent product should behave like a pinned, almost-permanent resident. A prefix generated once for a low-traffic RAG query should behave like a disposable scratch buffer. An LRU or LFU policy has no concept of \"this block will be reused by forty concurrent sessions in the next thirty seconds\" versus \"this block will never be touched again.\" That's not a caching decision. That's a resource allocation decision, and resource allocation is best modeled as a market: blocks have a price (expected future value per byte of memory held), and eviction should always take the lowest-priced block first, not the least-recently-touched one.\n\nYou don't need real auction mechanics to get most of the benefit — you need policy inputs that current systems mostly discard. Concretely:\n\n**Tag requests by workload class at admission time.** Agent loop, multi-turn chat, single-shot, RAG. This is metadata you already have at the API layer; it rarely makes it down to the cache manager.\n\n**Give shared prefixes a reference count, not just a timestamp.** A block referenced by many in-flight or recently-active sessions should be nearly eviction-proof regardless of recency.\n\n**Model decay differently per class.** A RAG block's value should decay to near-zero the moment the response is streamed. An agent block's value should decay slowly across a bounded idle window, because the next tool-call round trip is expected, not incidental.\n\n**Let admission control see cache pressure before accepting new work.** If the cache is full of high-value agent state, throttling new single-shot requests is often the right call — but only if the scheduler and the cache manager are actually talking to each other, which in most stacks they aren't.\n\nNone of this requires exotic infrastructure. It requires refusing to let the KV cache be a black box sitting underneath the scheduler. The moment you expose workload class and reuse expectation as first-class signals, eviction stops being a memory-pressure reflex and becomes a decision that's actually aligned with what your traffic looks like.\n\nThe industry's attention on inference economics has focused heavily on quantization and hardware — shrinking weights, picking the right accelerator, squeezing more tokens per dollar out of memory bandwidth. Those are real levers. But as more production traffic shifts toward agents and long-lived multi-turn sessions rather than stateless single-shot completions, the cost of a naive eviction policy grows faster than the cost of an unoptimized kernel. A bad quantization choice costs you accuracy at the margins. A bad eviction policy costs you cascading latency across unrelated requests, invisibly, and it gets worse under exactly the load patterns — bursty agent traffic, shared system prompts — that most teams are trying to scale into.\n\nThe fix isn't a smarter cache. It's admitting that \"cache\" was never the right word for what's sitting in that GPU memory. It's a scarce, shared, unevenly valuable resource being allocated under contention in real time. That's a market problem. Model it like one, and the latency wins show up in places you weren't even looking.", "url": "https://wpnews.pro/news/the-kv-cache-is-a-market-not-a-cache", "canonical_source": "https://dev.to/aiexplore369zoho/the-kv-cache-is-a-market-not-a-cache-801", "published_at": "2026-07-30 13:01:03+00:00", "updated_at": "2026-07-30 13:33:00.937085+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-agents", "ai-products"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/the-kv-cache-is-a-market-not-a-cache", "markdown": "https://wpnews.pro/news/the-kv-cache-is-a-market-not-a-cache.md", "text": "https://wpnews.pro/news/the-kv-cache-is-a-market-not-a-cache.txt", "jsonld": "https://wpnews.pro/news/the-kv-cache-is-a-market-not-a-cache.jsonld"}}