{"slug": "kv-cache-size-how-many-users-fit-on-one-gpu", "title": "KV Cache Size: How Many Users Fit on One GPU?", "summary": "A KV-cache calculator published on GitHub computes per-token cache size from a model's config.json using the formula 2 × layers × kv_heads × head_dim × bytes per value, showing that within 40 GiB of GPU memory reserved for cache and 4k-token conversations, Llama 3 8B fits 80 concurrent users at 128 KiB per token while DeepSeek V3 fits 149 at roughly 68.6 KiB per token. The analysis attributes the gap to grouped-query attention, which cuts Llama 3 8B's cache 4x from Llama 2 7B's 512 KiB per token, and to DeepSeek V3's multi-head latent attention, which stores one compressed vector of 512 latent plus 64 positional values per layer. Sliding-window layers in gpt-oss-20b stop growing after 128 tokens, costing about 24 KiB per extra token past that point only if the inference engine frees out-of-window cache.", "body_md": "DeepSeek V3 has 671 billion parameters. Its KV cache per token is roughly half the size of Llama 3 8B’s. That cache decides how many users fit on your GPU, and you can compute it from a config file in two minutes.\n\nSebastian Raschka’s [Big LLM Architecture Comparison](https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison) explains the design choices behind this: grouped-query attention, latent attention, sliding windows. I wanted to know what they cost in bytes, because that is the number on your GPU bill.\n\n**TL;DR**\n\nWhen an LLM writes a reply, each new token has to look back at every token before it. Recomputing all of that at every step would be far too slow, so the model keeps the intermediate results (the *keys* and *values*) in GPU memory and reuses them.\n\nThat saves compute. It costs memory. The cache grows with every token, and every user in the batch has their own copy.\n\nSo two things you control, context length and number of concurrent users, matter as much as the model’s parameter count.\n\nFor standard attention (multi-head or grouped-query):\n\n```\nbytes per token = 2 × layers × kv_heads × head_dim × bytes per value\n```\n\nThe 2 is for keys and values. Everything else comes from the model’s config.json.\n\nFor Llama 3 8B with a 16-bit cache, that is 131,072 bytes, or 128 KiB per token. A 4,000-token conversation holds about 0.5 GiB of cache. A hundred of them in parallel is about 49 GiB, before you have loaded a single weight.\n\nI ran the formula on five public models, using values from each model’s config or paper. The cache is 16-bit. The last column assumes you reserve 40 GiB of GPU memory for cache and every user has a 4k-token conversation, which every model here supports natively.\n\n† Beyond the model’s native context, so a what-if number. * gpt-oss sliding-window layers stop growing after 128 tokens, so past that point each extra token costs about 24 KiB. That only holds if your inference engine actually frees out-of-window cache. Check before you rely on it.\n\nIn the same 40 GiB, the full-attention design fits 20 conversations and Llama 3 8B fits 80. DeepSeek V3, roughly 80 to 100 times larger by parameter count, fits 149.\n\n**Grouped-query attention (GQA): share keys and values.** Llama 2 7B has 32 attention heads, and each keeps its own keys and values. Llama 3 8B keeps 32 query heads but only 8 sets of keys and values, so the cache falls from 512 to 128 KiB per token, exactly 4x.\n\n**Multi-head latent attention (MLA): compress them.** DeepSeek V3 does not store full keys and values. It stores one compressed vector per layer (512 latent values plus 64 positional ones) and rebuilds what it needs. That is 61 × 576 × 2 bytes, about 68.6 KiB per token, which is about half of Llama 3 8B’s cache.\n\n**Sliding-window attention: stop remembering everything.** Some layers only look at the most recent tokens. gpt-oss-20b alternates sliding-window layers (128 tokens) with full-attention layers, so half its layers can keep a fixed-size cache however long the conversation gets, if the engine frees the rest. Raschka covers the same idea in Gemma 3, which mixes local and global layers at a 5:1 ratio.\n\nThe gap widens as conversations get longer:\n\nSay these out loud if you reuse the numbers:\n\nThe core of the script is this:\n\n``` python\ndef per_token_bytes(layers, kv_heads, head_dim, bytes_per_value=2):    return 2 * layers * kv_heads * head_dim * bytes_per_value\nprint(per_token_bytes(32, 32, 128) / 1024)  # Llama 2 7B: 512.0 KiB per tokenprint(per_token_bytes(32, 8, 128) / 1024)   # Llama 3 8B: 128.0 KiB per token\n```\n\nThe full version, with MLA and sliding-window handling, is here: [*\\[GitHub link\\]*](https://github.com/Mehedi-Hasan-Abir/kv-cache-calculator.git). Copy a model’s config.json values into the config list and it prints the table for you.\n\nWhat does the number come out to for the model you run? Post it in the comments, and say which model and context length.\n\nNext: [How Many RAG Users Fit on One GPU?](LINK), using this math as the first filter.\n\n**LLM Architecture for People Who Ship:** Part 1: [Why One AI Model Can Serve 4x More Users on the Same GPU](https://medium.com/@mhabir102/why-one-ai-model-can-serve-4x-more-users-on-the-same-gpu-06d6387a9559?sharedUserId=mhabir102) · Part 2: [KV Cache Size: How Many Users Fit on One GPU?](https://medium.com/@mhabir102/kv-cache-size-how-many-users-fit-on-one-gpu-76bcd3c02ee9?sharedUserId=mhabir102) · Part 3: [How Many RAG Users Fit on One GPU?](https://medium.com/@mhabir102/how-many-rag-users-fit-on-one-gpu-a-five-step-check-4a347ee00019?sharedUserId=mhabir102)\n\n[KV Cache Size: How Many Users Fit on One GPU?](https://pub.towardsai.net/kv-cache-size-how-many-users-fit-on-one-gpu-76bcd3c02ee9) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/kv-cache-size-how-many-users-fit-on-one-gpu", "canonical_source": "https://pub.towardsai.net/kv-cache-size-how-many-users-fit-on-one-gpu-76bcd3c02ee9?source=rss----98111c9905da---4", "published_at": "2026-11-21 05:47:33+00:00", "updated_at": "2026-11-21 06:12:53.393104+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-research"], "entities": ["DeepSeek V3", "Llama 3 8B", "Llama 2 7B", "gpt-oss-20b", "Gemma 3", "Sebastian Raschka"], "also_reported_by": [], "takeaway": "Compute KV-cache memory as 2 × layers × kv_heads × head_dim × bytes per value: Llama 3 8B's grouped-query attention costs 128 KiB per token versus Llama 2 7B's 512 KiB, and DeepSeek V3's latent attention about 68.6 KiB, so 40 GiB of cache holds 80, 20 and 149 concurrent 4k-token conversations respec", "recommendation": {"advice": "Size GPU memory for concurrent users by computing per-token KV cache from config.json before deployment; in 40 GiB, Llama 3 8B fits 80 4k-token conversations and DeepSeek V3 fits 149.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/kv-cache-size-how-many-users-fit-on-one-gpu", "markdown": "https://wpnews.pro/news/kv-cache-size-how-many-users-fit-on-one-gpu.md", "text": "https://wpnews.pro/news/kv-cache-size-how-many-users-fit-on-one-gpu.txt", "jsonld": "https://wpnews.pro/news/kv-cache-size-how-many-users-fit-on-one-gpu.jsonld"}}