{"slug": "the-kv-cache-explained-why-long-conversations-get-expensive", "title": "The KV Cache Explained: Why Long Conversations Get Expensive", "summary": "The KV cache, which stores the keys and values of previously processed tokens so they are not recomputed, is the largest dynamic consumer of GPU memory in LLM serving systems and is what actually caps concurrency, according to an explainer on the topic. For a 70B-class model with 80 layers, 8 KV heads, and a head dimension of 128 in 16-bit precision, the cache costs about 320 KB per token, meaning 8,000 tokens of context for one user requires roughly 2.5 GB, and 100 users at that context length need about 250 GB—more than the model's 140 GB of weights. Techniques like PagedAttention store the cache in small fixed-size blocks allocated on demand to improve efficiency.", "body_md": "***TL;DR***\n\nGenerating each new token requires attention over every previous token. Without caching, the model would redo that work on every single step.\n\nThe keys and values of past tokens **never change**, so they are computed once and stored. That store is the KV cache.\n\nIts size is fixed per token by the model’s architecture, and it scales with **context length × concurrent users**. It routinely exceeds the size of the model weights.\n\nReserving memory per sequence up front wastes most of it, because you never know how long a response will be.\n\n**PagedAttention** stores the cache in small fixed-size blocks instead, allocated on demand — which is where a large share of modern serving throughput comes from.\n\nWhen the pool fills, the engine evicts running requests. Cache memory is what actually caps concurrency.\n\nAsk an LLM a question and the answer arrives one token at a time. To produce each token, the model attends to everything before it — the prompt and every token it has already generated.\n\nDone naively, that is a staggering amount of repeated work. Generating token 500 would mean reprocessing 499 tokens that were already processed to generate token 499, which had already reprocessed the 498 before it.\n\nThe **KV cache** is what stops that. It is a simple idea with outsized consequences: it is usually the largest dynamic consumer of GPU memory in a serving system, and it — not compute — is what decides how many users a GPU can hold.\n\nThis article covers what it stores, what it costs, how it is managed, and what happens when it runs out.\n\nA short primer, only as much as we need.\n\nFor every token it processes, a transformer produces three vectors:\n\n**Query (Q)** — what this token is looking for. **Key (K)** — what this token offers, so other tokens can find it. **Value (V)** — the actual information this token contributes.\n\nTo compute the output for a given token, the model compares that token’s query against the keys of all preceding tokens, and uses the resulting weights to blend their values.\n\nSo generating a new token requires **the K and V of every token before it.**\n\nHere is the property everything depends on.\n\nWhen the model generates token 501, it needs the keys and values of tokens 1 through 500. Those keys and values were already computed when those tokens were processed — and **they do not change.** Token 200’s key is a function of token 200 and the tokens before it. Adding new tokens to the end of the sequence does not alter it.\n\nThe queries are different: each new token needs a fresh query. But queries are needed only for the token being generated right now, and then discarded.\n\n```\nGenerating a new token needs:  Q  →  only for the current token   →  compute fresh, throw away  K  →  for every previous token     →  same as last step  V  →  for every previous token     →  same as last step\n```\n\nSo the keys and values get computed once and kept. That store is the **KV cache**.\n\nWithout it, every generation step reprocesses the entire context. With it, each step processes exactly one new token and reads the rest from memory.\n\nThe cache is not free, and its size is fixed by the model’s architecture. Per token:\n\n```\nbytes per token = 2 × layers × KV heads × head dimension × bytes per value\n```\n\nThe leading 2 is for K and V. Every layer keeps its own, because attention happens at every layer.\n\nFor a 70B-class model with 80 layers, 8 KV heads, a head dimension of 128, in 16-bit precision:\n\n```\n2 × 80 × 8 × 128 × 2 bytes ≈ 320 KB per token\n```\n\nThat looks small. Now scale it:\n\n```\n8,000 tokens of context  ≈   2.5 GB   for one user   100 users at 8,000 tokens  ≈   250 GB\n```\n\nCompare that with the model itself, which needs roughly 140 GB of weights at the same precision. **The cache for a hundred moderately-long conversations is larger than the model.**\n\nThis is the fact that surprises people. Weights are the number everyone sizes hardware on, and in a busy deployment they are frequently the smaller half of the memory budget.\n\nThe cache grows along two dimensions at once, and both are outside your control.\n\n**Context length.** Every token added to a conversation adds its slice. A conversation that has been running for a while costs more than a fresh one, even from the same user.\n\n**Concurrency.** Every simultaneous request has its own cache. They do not share anything.\n\nMultiply them and you get the real number. This is why a deployment can be perfectly stable at a hundred short conversations and fall over at twenty long ones, on identical hardware with identical compute.\n\nIt also means capacity questions are unanswerable in the abstract. “How many users can this GPU serve?” has no answer without a context length attached to it.\n\nSo we know what to store. The harder question is *where* — and this is where naive implementations lose most of their memory.\n\nThe difficulty: **you do not know how long a sequence will be.** A request arrives with a 200-token prompt. Will the answer be 5 tokens or 5,000? Nobody knows, including the model.\n\nEarly serving systems handled this by reserving a contiguous region large enough for the maximum supported length, for every sequence. If the model supports 32,000 tokens, every request reserves 32,000 tokens’ worth of cache the moment it is admitted.\n\nThe waste is severe, and it comes in three forms:\n\n**Over-reservation.** A request that generates 100 tokens holds a 32,000-token reservation for its entire life. Over 99% of that memory is untouched but unavailable to anyone else.\n\n**Internal fragmentation.** Even when a sequence does grow, the unused tail of its reservation sits idle until it gets there — which it may never do.\n\n**External fragmentation.** Requests finish in a different order than they started, leaving gaps of free memory between live reservations. The gaps may add up to plenty of space, but if none is large enough to hold a new contiguous reservation, a new request cannot be admitted.\n\nThe vLLM paper that introduced PagedAttention reported that systems of this kind wasted the large majority of their KV cache memory. Which meant the practical limit on concurrency was not the hardware — it was the allocator.\n\nThe fix is to abandon the requirement that a sequence’s cache be contiguous.\n\n**PagedAttention** splits the cache into small fixed-size **blocks**, each holding a fixed number of tokens (16 is a common default). A sequence is allocated blocks as it grows, one at a time, and those blocks can sit anywhere in the pool. A per-sequence **block table** records which physical blocks hold which part of the sequence, and the attention kernel follows that table when it reads.\n\nThe technique is borrowed from operating system virtual memory, which is where the name comes from — pages, page tables, and a layer of indirection between logical and physical addresses.\n\nThree things change immediately:\n\n**No over-reservation.** A sequence holds only the blocks it has actually filled. A 100-token answer uses seven blocks, not two thousand.\n\n**No external fragmentation.** Every block is the same size, so any free block fits any sequence. Free memory is never unusable because of its shape.\n\n**Waste is bounded.** The only unused space is the partially-filled last block of each sequence — at most 15 tokens’ worth. Compared to reserving thousands, this is nothing.\n\nThe practical effect is that far more sequences fit in the same VRAM. Since concurrency is what drives batching, and batching is what drives throughput, better cache allocation translates fairly directly into more tokens per second from the same GPU.\n\nBlocks are allocated on demand, which raises an obvious question: what happens when a running sequence needs a new block and none is free?\n\nThe engine has to take memory from someone. It **preempts** a running request, freeing its blocks, and resumes it later when space is available. Two ways to do that:\n\n**Recompute.** Discard the sequence’s cache entirely. When it resumes, run prefill again over its tokens to rebuild the state. Cheap to evict, expensive to restore.\n\n**Swap.** Copy the blocks out to CPU memory and copy them back on resume. Preserves the work, but pays the cost of moving data across the PCIe link in both directions.\n\nEither way, from the user’s side this appears as a response that stalls mid-generation and then continues. From the operator’s side, frequent preemption is a clear signal that the deployment is running past its memory limits — usually the point to reduce concurrency, shorten the maximum context, or add capacity.\n\nThis is also the mechanism behind a claim that sounds odd until you know where it comes from: **concurrency in an LLM server is capped by cache memory, not by compute.** The GPU is rarely out of arithmetic. It is out of blocks.\n\nSince cache size is set by architecture and context, there are only a few levers.\n\n**Grouped-query attention (GQA)** is the big one, and it is why the number in Section 3 was as small as it was. In classic multi-head attention, every attention head keeps its own K and V. GQA has several query heads share one set, cutting the KV heads dramatically. That 70B example used 8 KV heads; with 64 it would have been eight times larger — 2.5 MB per token instead of 320 KB. Nearly every recent model uses GQA or a variant, and it is largely a serving-cost decision.\n\n**KV cache quantization** stores keys and values in 8-bit instead of 16-bit, halving the cache at some cost to quality. Independent of weight quantization — you can do either, or both.\n\n**Context limits.** Capping the maximum accepted context bounds what any single request can consume, protecting everyone else’s capacity. Blunt, but effective and entirely under your control.\n\n**Longer-term architectural work** — sliding-window attention, sparse and linear attention, cross-layer sharing — aims at changing the growth curve rather than its constant. This is an active area and worth watching if long context matters to your workload.\n\nFour sentences hold most of it:\n\nThe weights are static, shared by everyone, and sized once. The KV cache is dynamic, private to each request, and grows with both context length and concurrency. Whatever VRAM the weights do not occupy becomes the cache pool. **How efficiently that pool is managed determines how many users the GPU can serve.**\n\nThat is why memory management is not a detail of LLM serving. It is most of it.\n\nThere is one thing we have quietly accepted throughout.\n\nWhen a request finishes, its cache is released. The blocks return to the pool and their contents are gone. That is correct — the conversation is over, the state is not needed.\n\nBut look at what actually flows through a production deployment. Every request from the same application begins with the same system prompt, often thousands of tokens of instructions and policy. Every follow-up turn in a chat re-sends the entire conversation so far. Users in the same product ask questions against the same documents.\n\nThe prefixes are identical. And we compute them from scratch, every single time, throwing the result away at the end of each request.\n\nThat is a lot of work being repeated — the very thing the KV cache was invented to avoid, reappearing one level up. Within a request we already refuse to recompute. Across requests, we do nothing.\n\n**If two requests start with the same tokens, why compute the same cache twice?**\n\nThat question turns out to have a genuinely interesting answer.\n\n**1. What really happens when you click ‘Send’ on ChatGPT** — A journey through modern AI Infrastructure\n\n**2. What Do You Do With a Model That’s Too Big for Your GPU?** — Quantization, Sharding and Parallelism Explained\n\n**3. How Does One GPU Serve Hundreds of Users at the Same Time?** — Inside an LLM inference server\n\n**4. The KV Cache Explained: Why Long Conversations Get Expensive** — How LLMs remember context without recomputing everything\n\n**5. Why Is Your LLM Recomputing the Same Prompt 1,000 Times a Day?** — Prefix caching, radix trees and block hashing explained\n\n(Next Article)\n\n**6. Why Traditional Load Balancing Breaks for LLMs** — Building an LLM-aware router\n\n**7. Kubernetes for LLM Inference: How AI Workloads Run Across a GPU Cluster**\n\n**8. LLM-D Explained** — How modern AI infrastructure routes, schedules and scales LLM inference\n\n**9. Inside a Modern AI Inference Platform** — The full stack end-to-end\n\n**Sources**\n\n[The KV Cache Explained: Why Long Conversations Get Expensive](https://pub.towardsai.net/the-kv-cache-explained-why-long-conversations-get-expensive-4bd8f77dd7e7) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/the-kv-cache-explained-why-long-conversations-get-expensive", "canonical_source": "https://pub.towardsai.net/the-kv-cache-explained-why-long-conversations-get-expensive-4bd8f77dd7e7?source=rss----98111c9905da---4", "published_at": "2026-09-07 16:31:00+00:00", "updated_at": "2026-09-07 16:55:03.028660+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/the-kv-cache-explained-why-long-conversations-get-expensive", "markdown": "https://wpnews.pro/news/the-kv-cache-explained-why-long-conversations-get-expensive.md", "text": "https://wpnews.pro/news/the-kv-cache-explained-why-long-conversations-get-expensive.txt", "jsonld": "https://wpnews.pro/news/the-kv-cache-explained-why-long-conversations-get-expensive.jsonld"}}