The KV Cache Explained: Why Long Conversations Get Expensive

The KV cache, which stores the keys and values of previously processed tokens so they are not recomputed, is the largest dynamic consumer of GPU memory in LLM serving systems and is what actually caps concurrency, according to an explainer on the topic. For a 70B-class model with 80 layers, 8 KV heads, and a head dimension of 128 in 16-bit precision, the cache costs about 320 KB per token, meaning 8,000 tokens of context for one user requires roughly 2.5 GB, and 100 users at that context length need about 250 GB—more than the model's 140 GB of weights. Techniques like PagedAttention store the cache in small fixed-size blocks allocated on demand to improve efficiency.

TL;DR Generating each new token requires attention over every previous token. Without caching, the model would redo that work on every single step. The keys and values of past tokens never change , so they are computed once and stored. That store is the KV cache. Its size is fixed per token by the model’s architecture, and it scales with context length × concurrent users . It routinely exceeds the size of the model weights. Reserving memory per sequence up front wastes most of it, because you never know how long a response will be. PagedAttention stores the cache in small fixed-size blocks instead, allocated on demand — which is where a large share of modern serving throughput comes from. When the pool fills, the engine evicts running requests. Cache memory is what actually caps concurrency. Ask an LLM a question and the answer arrives one token at a time. To produce each token, the model attends to everything before it — the prompt and every token it has already generated. Done naively, that is a staggering amount of repeated work. Generating token 500 would mean reprocessing 499 tokens that were already processed to generate token 499, which had already reprocessed the 498 before it. The KV cache is what stops that. It is a simple idea with outsized consequences: it is usually the largest dynamic consumer of GPU memory in a serving system, and it — not compute — is what decides how many users a GPU can hold. This article covers what it stores, what it costs, how it is managed, and what happens when it runs out. A short primer, only as much as we need. For every token it processes, a transformer produces three vectors: Query Q — what this token is looking for. Key K — what this token offers, so other tokens can find it. Value V — the actual information this token contributes. To compute the output for a given token, the model compares that token’s query against the keys of all preceding tokens, and uses the resulting weights to blend their values. So generating a new token requires the K and V of every token before it. Here is the property everything depends on. When the model generates token 501, it needs the keys and values of tokens 1 through 500. Those keys and values were already computed when those tokens were processed — and they do not change. Token 200’s key is a function of token 200 and the tokens before it. Adding new tokens to the end of the sequence does not alter it. The queries are different: each new token needs a fresh query. But queries are needed only for the token being generated right now, and then discarded. Generating a new token needs: Q → only for the current token → compute fresh, throw away K → for every previous token → same as last step V → for every previous token → same as last step So the keys and values get computed once and kept. That store is the KV cache . Without it, every generation step reprocesses the entire context. With it, each step processes exactly one new token and reads the rest from memory. The cache is not free, and its size is fixed by the model’s architecture. Per token: bytes per token = 2 × layers × KV heads × head dimension × bytes per value The leading 2 is for K and V. Every layer keeps its own, because attention happens at every layer. For a 70B-class model with 80 layers, 8 KV heads, a head dimension of 128, in 16-bit precision: 2 × 80 × 8 × 128 × 2 bytes ≈ 320 KB per token That looks small. Now scale it: 8,000 tokens of context ≈ 2.5 GB for one user 100 users at 8,000 tokens ≈ 250 GB Compare that with the model itself, which needs roughly 140 GB of weights at the same precision. The cache for a hundred moderately-long conversations is larger than the model. This is the fact that surprises people. Weights are the number everyone sizes hardware on, and in a busy deployment they are frequently the smaller half of the memory budget. The cache grows along two dimensions at once, and both are outside your control. Context length. Every token added to a conversation adds its slice. A conversation that has been running for a while costs more than a fresh one, even from the same user. Concurrency. Every simultaneous request has its own cache. They do not share anything. Multiply them and you get the real number. This is why a deployment can be perfectly stable at a hundred short conversations and fall over at twenty long ones, on identical hardware with identical compute. It also means capacity questions are unanswerable in the abstract. “How many users can this GPU serve?” has no answer without a context length attached to it. So we know what to store. The harder question is where — and this is where naive implementations lose most of their memory. The difficulty: you do not know how long a sequence will be. A request arrives with a 200-token prompt. Will the answer be 5 tokens or 5,000? Nobody knows, including the model. Early serving systems handled this by reserving a contiguous region large enough for the maximum supported length, for every sequence. If the model supports 32,000 tokens, every request reserves 32,000 tokens’ worth of cache the moment it is admitted. The waste is severe, and it comes in three forms: Over-reservation. A request that generates 100 tokens holds a 32,000-token reservation for its entire life. Over 99% of that memory is untouched but unavailable to anyone else. Internal fragmentation. Even when a sequence does grow, the unused tail of its reservation sits idle until it gets there — which it may never do. External fragmentation. Requests finish in a different order than they started, leaving gaps of free memory between live reservations. The gaps may add up to plenty of space, but if none is large enough to hold a new contiguous reservation, a new request cannot be admitted. The vLLM paper that introduced PagedAttention reported that systems of this kind wasted the large majority of their KV cache memory. Which meant the practical limit on concurrency was not the hardware — it was the allocator. The fix is to abandon the requirement that a sequence’s cache be contiguous. PagedAttention splits the cache into small fixed-size blocks , each holding a fixed number of tokens 16 is a common default . A sequence is allocated blocks as it grows, one at a time, and those blocks can sit anywhere in the pool. A per-sequence block table records which physical blocks hold which part of the sequence, and the attention kernel follows that table when it reads. The technique is borrowed from operating system virtual memory, which is where the name comes from — pages, page tables, and a layer of indirection between logical and physical addresses. Three things change immediately: No over-reservation. A sequence holds only the blocks it has actually filled. A 100-token answer uses seven blocks, not two thousand. No external fragmentation. Every block is the same size, so any free block fits any sequence. Free memory is never unusable because of its shape. Waste is bounded. The only unused space is the partially-filled last block of each sequence — at most 15 tokens’ worth. Compared to reserving thousands, this is nothing. The practical effect is that far more sequences fit in the same VRAM. Since concurrency is what drives batching, and batching is what drives throughput, better cache allocation translates fairly directly into more tokens per second from the same GPU. Blocks are allocated on demand, which raises an obvious question: what happens when a running sequence needs a new block and none is free? The engine has to take memory from someone. It preempts a running request, freeing its blocks, and resumes it later when space is available. Two ways to do that: Recompute. Discard the sequence’s cache entirely. When it resumes, run prefill again over its tokens to rebuild the state. Cheap to evict, expensive to restore. Swap. Copy the blocks out to CPU memory and copy them back on resume. Preserves the work, but pays the cost of moving data across the PCIe link in both directions. Either way, from the user’s side this appears as a response that stalls mid-generation and then continues. From the operator’s side, frequent preemption is a clear signal that the deployment is running past its memory limits — usually the point to reduce concurrency, shorten the maximum context, or add capacity. This is also the mechanism behind a claim that sounds odd until you know where it comes from: concurrency in an LLM server is capped by cache memory, not by compute. The GPU is rarely out of arithmetic. It is out of blocks. Since cache size is set by architecture and context, there are only a few levers. Grouped-query attention GQA is the big one, and it is why the number in Section 3 was as small as it was. In classic multi-head attention, every attention head keeps its own K and V. GQA has several query heads share one set, cutting the KV heads dramatically. That 70B example used 8 KV heads; with 64 it would have been eight times larger — 2.5 MB per token instead of 320 KB. Nearly every recent model uses GQA or a variant, and it is largely a serving-cost decision. KV cache quantization stores keys and values in 8-bit instead of 16-bit, halving the cache at some cost to quality. Independent of weight quantization — you can do either, or both. Context limits. Capping the maximum accepted context bounds what any single request can consume, protecting everyone else’s capacity. Blunt, but effective and entirely under your control. Longer-term architectural work — sliding-window attention, sparse and linear attention, cross-layer sharing — aims at changing the growth curve rather than its constant. This is an active area and worth watching if long context matters to your workload. Four sentences hold most of it: The weights are static, shared by everyone, and sized once. The KV cache is dynamic, private to each request, and grows with both context length and concurrency. Whatever VRAM the weights do not occupy becomes the cache pool. How efficiently that pool is managed determines how many users the GPU can serve. That is why memory management is not a detail of LLM serving. It is most of it. There is one thing we have quietly accepted throughout. When a request finishes, its cache is released. The blocks return to the pool and their contents are gone. That is correct — the conversation is over, the state is not needed. But look at what actually flows through a production deployment. Every request from the same application begins with the same system prompt, often thousands of tokens of instructions and policy. Every follow-up turn in a chat re-sends the entire conversation so far. Users in the same product ask questions against the same documents. The prefixes are identical. And we compute them from scratch, every single time, throwing the result away at the end of each request. That is a lot of work being repeated — the very thing the KV cache was invented to avoid, reappearing one level up. Within a request we already refuse to recompute. Across requests, we do nothing. If two requests start with the same tokens, why compute the same cache twice? That question turns out to have a genuinely interesting answer. 1. What really happens when you click ‘Send’ on ChatGPT — A journey through modern AI Infrastructure 2. What Do You Do With a Model That’s Too Big for Your GPU? — Quantization, Sharding and Parallelism Explained 3. How Does One GPU Serve Hundreds of Users at the Same Time? — Inside an LLM inference server 4. The KV Cache Explained: Why Long Conversations Get Expensive — How LLMs remember context without recomputing everything 5. Why Is Your LLM Recomputing the Same Prompt 1,000 Times a Day? — Prefix caching, radix trees and block hashing explained Next Article 6. Why Traditional Load Balancing Breaks for LLMs — Building an LLM-aware router 7. Kubernetes for LLM Inference: How AI Workloads Run Across a GPU Cluster 8. LLM-D Explained — How modern AI infrastructure routes, schedules and scales LLM inference 9. Inside a Modern AI Inference Platform — The full stack end-to-end Sources The KV Cache Explained: Why Long Conversations Get Expensive https://pub.towardsai.net/the-kv-cache-explained-why-long-conversations-get-expensive-4bd8f77dd7e7 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.