Your KV Cache Is Bigger Than Your Model OpenAI's gpt-oss-120b model, with open weights, requires 72 KiB of KV cache per token in 16-bit precision, calculated from its config.json with 36 layers, 8 key-value heads, and a head dimension of 64. The KV cache stores two vectors per token per layer, and its size grows linearly with sequence length, making it a primary factor in capacity planning for LLM inference. This is Part two of seven. The layer where capacity planning actually happens. Part 1 : Start Here: The Words Everyone Uses About LLM Inference https://medium.com/towards-artificial-intelligence/start-here-the-words-everyone-uses-about-llm-inference-f4f1edc4a1b2 Part one ended on a problem it could not solve. Batching lets many users share one read of the model’s weights, which is what makes serving affordable, but the part of the model that remembers your conversation is yours alone, and no amount of batching spreads that cost. This part is about how big that private memory actually gets. Capacity plans come apart here more than anywhere else in the stack, and they fail the same way every time: the model fits, the demo is fast, and then real concurrency arrives. When a model reads text, each token produces two vectors that attention needs later: a key and a value . Rather than recompute them for every previous token each time it writes a new token, the model stores them. That store is the KV cache . It is a straight trade of memory for time. Without it, every step would have to recompute the key and value for every token that came before; with it, every step computes exactly one. That is the saving, and it is a per-step saving: work that would grow with the conversation on every single step instead stays flat. It is also the reason a long conversation costs more than a short one even when the reply is identical. It is not storing your text. The name suggests a transcript, and the mental model that follows from that is wrong in a way that costs money. What is stored is two vectors per token per layer: numbers, not characters. And they are not interchangeable between two phrasings that mean the same thing. Take two customers asking a support assistant the same question: Five tokens against six. Four of them are identical and none of them are reusable, because matching starts at the first token and stops at the first difference. What's is a single token; What is is two. The sequences diverge at the very first token, so the shared prefix is zero, despite four identical tokens sitting right there in the middle. A token's key and value depend on its position and on every token before it, so your at position 1 is a different entry from your at position 2. Reuse is exact-match on a token sequence, counted from the first token. Not on meaning, and not on anything a humanwould call the same question. And the split above belongs to this tokenizer. Run the same two strings through GPT-4’s encoding and What's becomes two tokens rather than one, which moves the divergence from position zero to position one and hands you a shared token you did not have. Llama, Qwen, Mistral and Gemma each split differently again. Whatever you measure this with has to be the tokenizer the model you serve actually uses, or you have measured somebody else's deployment. Two things at once, and the second is the one people drop. With a cache you compute one key-value pair per step instead of all of them. But the cache is still read in full on every step. So the cost stopped growing like a curve and started growing like a straight line, which is not the same as becoming small. A key and a value, times the number of layers, times the number of key-value heads, times the head dimension, times the bytes per number. Five numbers with four multiplication signs between them, which is where the heading comes from. Three of the five you read straight out of the model’s config file. One is the constant 2, because every token stores a key and a value. The last is your choice of precision. Take gpt-oss-120b , which has open weights, so every number below can be read out of its config.json with no account and no gate. It has 36 layers, 8 key-value heads and a head dimension of 64. In 16-bit that comes to 72 KiB for every token. Then the architecture hands half of it back, and this is the part worth reading a config for. The same file has "sliding window": 128 and a layer types list that alternates. Half the layers attend to the whole conversation. The other half only ever look at the last 128 tokens. The fifth factor is the one that bites. num hidden layers says 36 and layer types says only 18 of them grow, so the same config supports two answers that differ by exactly a factor of two. Eighteen full-attention layers grow with the conversation, at 36 KiB a token. The eighteen sliding-window layers hold 128 tokens between them and stop: 4.5 MiB, fixed, however long it runs. Every unit is drawn rather than summarised: 36 layer bars, 64 query ticks over 8 key-value boxes, 128 expert cells with four alight. Count anything in it and you get the config back. So the honest figure is 36 KiB per token, provided your engine allocates the sliding layers as sliding layers. If it does not, you are back to 72 KiB and half the capacity, which is worth checking before you plan around this. Multiply out: Plus the fixed 4.5 MiB from the sliding half, on top of every row. It is 3% at 4K and rounding error at 128K, which is why the rest of this article works from the growing part alone. That is a recent architecture working hard at this problem. Run the identical formula over the shape most people still picture 80 layers, 8 key-value heads, head dimension 128 and you get 320 KiB per token and 40 GiB at that same 128K. Same five numbers, same multiplication, and a factor of nearly nine between two real models. Which is why the formula is worth more to you than any single figure quoted from it. And it is not only capacity. Attention re-reads the cache to produce every token, so at 4.5 GiB and 3.35 TB/s of bandwidth, reading it costs about 1.4 milliseconds per token before any arithmetic happens at all. That is for one user, doing nothing clever. Part one described that roof in the abstract; this is what it costs in milliseconds. Here is the calculation that belongs in every capacity plan and is almost never in one, and it is worth going slowly, because every number after it divides by the answer. Take two H100s under tensor parallelism, so the cache is one pool rather than two: 160 GB. You do not get all of it. vLLM’s --gpu-memory-utilization defaults to 0.92, so call it 147 GB. gpt-oss-120b's fifteen weight shards sum to 65.25 GB its expert weights ship in MXFP4… Attention, router and embeddings do not . That leaves 82 GB, which is 76 GiB , and once activations and CUDA graph buffers are out you are at roughly 71 GiB for KV cache . One card would have left about 3 GiB after the same deductions, not enough for a single 128K conversation. So the second card is bought for the cache, not for the model. That is the argument of this entire article, arriving about twenty paragraphs early. Divide by the per-conversation cost: The budget stays full throughout. What changes is how few tenants fit inside it. Five hundred and four conversations at 4K. Two hundred and fifty-two at 8K. Sixty-three at 32K. Fifteen at 128K. Those are ceilings, and worth reading as such: they charge every conversation its full context length. Paging, a few sections down, is exactly what stops you paying that: a conversation occupies what it has actually used. The ceiling is what you plan for; the live number floats below it. Now hold that against part one, where the ridge was 296. At 4K you clear it comfortably, 504 against 296. At 8K you fall just short at 252. At 32K you are at a fifth of it. At 128K, a twentieth. Same card, same model, same weights. How long the conversations are decides whether you get to use the arithmetic you paid for. Which turns the title of this article into a number. Fifteen conversations at 128K is 67.5 GiB of cache, or 72.5 GB against 65.25 GB of weights. One conversation’s cache is nothing next to the model. A card full of them outweighs it. The crossover lands at about 8K. That is not a long context by 2026 standards. It is a chat with some history in it, or a document of a few pages. At that point the cache stops just costing you memory and starts costing you the batch size, which was the whole mechanism part one described. There is a second bite, specific to this model. gpt-oss-120b is a mixture of experts and fires 4 of its 128 experts per token, so its expert layers see an intensity of the batch times 0.031, and reaching the ridge there would take a batch near 9,500. Even at 504 conversations the expert layers stay bandwidth-bound — and they are 93% of the bytes fetched per step, not half of them. The small cache lets you batch. The sparsity then takes most of the winnings back. Part one flagged this as a footnote; here it is with a real model attached. Not storing it in the first place. The cheapest saving available is a layer that never needs the history. gpt-oss alternates full attention with layers that only ever look at the last 128 tokens, and those layers stop growing the moment a conversation passes the window — half the layers, a fixed 4.5 MiB between them, however long it runs. The cost is real and worth saying plainly: those layers cannot see the beginning of your conversation, and every long-range dependency in the model has to travel through the other half. Sharing key-value heads. In the original design every query head had its own key and value heads. Grouped-query attention shares them: gpt-oss-120b has 64 query heads and 8 key-value heads, an eightfold cut. Run the formula with 64 instead of 8 and that 4.5 GiB becomes 36 GiB for a single conversation. Long context would simply not exist as a feature. This is not a tuning knob. It is the thing that makes long context a product rather than a demo. Compressing before storing. Multi-head latent attention , the approach DeepSeek uses, stores a small compressed vector and expands it back when needed. A bigger saving than grouped-query attention, paid for with extra arithmetic — which, given everything in part one, is exactly the right currency to pay in. Paging it. The obvious implementation reserves a contiguous block per conversation, sized for the longest it might get. Most never get there, so you have bought memory nobody uses. PagedAttention , which is vLLM’s contribution, stores the cache in fixed-size blocks with a lookup table, exactly like virtual memory in an operating system. Blocks get allocated as needed, and two conversations that begin identically can point at the same physical block — which turns out to matter more than the fragmentation saving. Reusing shared prefixes. This is what that block sharing buys. If a thousand requests share a 2,000-token system prompt, you process it once. The case that matters most is quieter, though: in any multi-turn chat, turn two contains the whole of turn one. A second question appended to a 24-token exchange reuses 23 of the 45 tokens it now sends. The conversation is the shared prefix, and it grows every turn. Not 24, incidentally: turn one ended on ? as its own token, and in turn two the newline after it merges the two into ?\n. The boundary token re-tokenizes, so you lose one. That is the level at which this matching works. On agentic workloads, few-shot prompts and any chat product with a long set of instructions at the start, this is often the biggest single win available, and it is a configuration flag rather than a project. It also has a production trap in it, and it is the most useful thing in this article for anyone whose prompts are templated. Anything that changes at the front kills reuse for everything behind it, because matching stops at the first token that differs. Thirteen per cent becomes ninety-six, for a template change. Session ids, timestamps, user names and request counters all belong after the stable part of your prompt, not before it. In practice the difference is sharper, because an engine does not reuse single tokens. vLLM matches in blocks, sixteen tokens by default, so a six-token match is zero blocks and no reuse at all , and a forty-three-token match is two blocks, or thirty-two tokens. Rounding down to the block boundary turns 13% into nothing and 96% into 71%. The advice does not change. The gap it closes gets wider. Measured with o200k base, the encoding gpt-oss uses. Same words, same 45 tokens, and only the timestamp moved. The second line under each count is the number that survives block alignment, which is the one your engine actually acts on. It is also the quickest way to publish a misleading benchmark, and part seven is about that at length. The short version: if a throughput number does not say whether prefix caching was on, it is not telling you about the model. Storing it in fewer bytes. Halving the cache to 8-bit roughly doubles the conversations that fit. Very cheap in quality terms, and it is one flag — --kv-cache-dtype fp8 — so it is the first thing I would try. One caveat that part three explains: not every attention backend supports an FP8 cache, and the ones that do not fall back silently to a slower path. Check the backend line after you turn it on, not just the memory. Nothing accumulates, incidentally — each entry is quantized once when it is written and never touched again — but the error it carries is present in every subsequent read of that entry, which is a lot of reads. At 8 bits that error is small enough not to matter. At 4 bits it is not, and that is part four. The gold bar marks the three that are yours today. The other three are decided when you pick the model, which is a different meeting and a longer lead time. Compute the number for your own model. Pull num key value heads, head dim and the layer count out of config.json and multiply. If you would rather not: the calculators run in a browser https://netsatsawat.github.io/llm-inference-arithmetic/ , and lia model