# Why the KV cache, not the weights, decides how many users an LLM server can hold > Source: > Published: 2026-10-25 11:37:15+00:00 A 7B model in fp16 takes 13.5 GB of GPU memory for its weights. On an 80 GB card that leaves about 66 GB free, which sounds like plenty. It runs out at around 30 users with 4,096-token contexts, and the weights aren't what fills it. The KV cache is. This post works through the arithmetic. All the numbers are for a Llama-2-7B-shaped model: 32 layers, 32 attention heads, head dimension 128, fp16. A decoder-only model generates one token per forward pass. The new token's query has to attend to the keys and values of every earlier token. Without a cache, each pass recomputes those keys and values for the whole prefix. The causal mask is what makes reuse safe. Earlier tokens can't see later ones, so appending a token never changes the keys and values already computed. The model can compute them once, store them, and on each step add one new key row and one new value row per layer. The saving is large. With a 5-token prompt and 64 generated tokens, recomputing everything costs 2,336 key/value projections per layer. With a cache it costs 68, about 34 times less. Attention still reads every cached key at each step, so the cache removes the recomputed projections, not the attention cost itself. The cache for one token is: ``` 2 (K and V) × layers × kv_heads × head_dim × bytes = 2 × 32 × 32 × 128 × 2 = 524,288 bytes ≈ 0.5 MiB ``` Half a megabyte per token adds up quickly: | Context | Batch | KV cache | |---|---|---| | 4,096 | 1 | 2.1 GB | | 32,768 | 1 | 17.2 GB | | 4,096 | 32 | 68.7 GB | A single 32K-token conversation needs more memory for its cache than the model needs for its weights. And the weights are paid once, while the cache is paid per sequence. So the number of users a server can batch is roughly the free memory divided by the cache for one sequence: 66.5 GB / 2.1 GB ≈ 30 sequences of 4,096 tokens, before activations and runtime buffers. Generation has two phases. Prefill runs the whole prompt through the model at once, so each byte of weights read from memory does work for every prompt token. Decode produces one token per step, and each byte of weights read does work for just that one token. Decode is limited by memory bandwidth, not by compute. Batching is the fix: one read of the weights serves every sequence in the batch. But every extra sequence brings its own KV cache, so cache size is what caps the batch, and with it the throughput. Each technique cuts one term of the formula: The first and third are architecture choices made before or during training. Cache quantization and paging can be applied at serving time. I built an interactive version of this as part of Lensa ML, a free set of ML explainers. The [KV cache explainer](https://lensaml.com/explainers/kv-cache) lets you step through generation token by token, drag the context length and batch size to watch the cache outgrow the weights, and toggle KV heads, cache precision and a sliding window to see each saving.