KV Cache Size: How Many Users Fit on One GPU? A KV-cache calculator published on GitHub computes per-token cache size from a model's config.json using the formula 2 × layers × kv_heads × head_dim × bytes per value, showing that within 40 GiB of GPU memory reserved for cache and 4k-token conversations, Llama 3 8B fits 80 concurrent users at 128 KiB per token while DeepSeek V3 fits 149 at roughly 68.6 KiB per token. The analysis attributes the gap to grouped-query attention, which cuts Llama 3 8B's cache 4x from Llama 2 7B's 512 KiB per token, and to DeepSeek V3's multi-head latent attention, which stores one compressed vector of 512 latent plus 64 positional values per layer. Sliding-window layers in gpt-oss-20b stop growing after 128 tokens, costing about 24 KiB per extra token past that point only if the inference engine frees out-of-window cache. DeepSeek V3 has 671 billion parameters. Its KV cache per token is roughly half the size of Llama 3 8B’s. That cache decides how many users fit on your GPU, and you can compute it from a config file in two minutes. Sebastian Raschka’s Big LLM Architecture Comparison https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison explains the design choices behind this: grouped-query attention, latent attention, sliding windows. I wanted to know what they cost in bytes, because that is the number on your GPU bill. TL;DR When an LLM writes a reply, each new token has to look back at every token before it. Recomputing all of that at every step would be far too slow, so the model keeps the intermediate results the keys and values in GPU memory and reuses them. That saves compute. It costs memory. The cache grows with every token, and every user in the batch has their own copy. So two things you control, context length and number of concurrent users, matter as much as the model’s parameter count. For standard attention multi-head or grouped-query : bytes per token = 2 × layers × kv heads × head dim × bytes per value The 2 is for keys and values. Everything else comes from the model’s config.json. For Llama 3 8B with a 16-bit cache, that is 131,072 bytes, or 128 KiB per token. A 4,000-token conversation holds about 0.5 GiB of cache. A hundred of them in parallel is about 49 GiB, before you have loaded a single weight. I ran the formula on five public models, using values from each model’s config or paper. The cache is 16-bit. The last column assumes you reserve 40 GiB of GPU memory for cache and every user has a 4k-token conversation, which every model here supports natively. † Beyond the model’s native context, so a what-if number. gpt-oss sliding-window layers stop growing after 128 tokens, so past that point each extra token costs about 24 KiB. That only holds if your inference engine actually frees out-of-window cache. Check before you rely on it. In the same 40 GiB, the full-attention design fits 20 conversations and Llama 3 8B fits 80. DeepSeek V3, roughly 80 to 100 times larger by parameter count, fits 149. Grouped-query attention GQA : share keys and values. Llama 2 7B has 32 attention heads, and each keeps its own keys and values. Llama 3 8B keeps 32 query heads but only 8 sets of keys and values, so the cache falls from 512 to 128 KiB per token, exactly 4x. Multi-head latent attention MLA : compress them. DeepSeek V3 does not store full keys and values. It stores one compressed vector per layer 512 latent values plus 64 positional ones and rebuilds what it needs. That is 61 × 576 × 2 bytes, about 68.6 KiB per token, which is about half of Llama 3 8B’s cache. Sliding-window attention: stop remembering everything. Some layers only look at the most recent tokens. gpt-oss-20b alternates sliding-window layers 128 tokens with full-attention layers, so half its layers can keep a fixed-size cache however long the conversation gets, if the engine frees the rest. Raschka covers the same idea in Gemma 3, which mixes local and global layers at a 5:1 ratio. The gap widens as conversations get longer: Say these out loud if you reuse the numbers: The core of the script is this: python def per token bytes layers, kv heads, head dim, bytes per value=2 : return 2 layers kv heads head dim bytes per value print per token bytes 32, 32, 128 / 1024 Llama 2 7B: 512.0 KiB per tokenprint per token bytes 32, 8, 128 / 1024 Llama 3 8B: 128.0 KiB per token The full version, with MLA and sliding-window handling, is here: \ GitHub link\ https://github.com/Mehedi-Hasan-Abir/kv-cache-calculator.git . Copy a model’s config.json values into the config list and it prints the table for you. What does the number come out to for the model you run? Post it in the comments, and say which model and context length. Next: How Many RAG Users Fit on One GPU? LINK , using this math as the first filter. LLM Architecture for People Who Ship: Part 1: Why One AI Model Can Serve 4x More Users on the Same GPU https://medium.com/@mhabir102/why-one-ai-model-can-serve-4x-more-users-on-the-same-gpu-06d6387a9559?sharedUserId=mhabir102 · Part 2: KV Cache Size: How Many Users Fit on One GPU? https://medium.com/@mhabir102/kv-cache-size-how-many-users-fit-on-one-gpu-76bcd3c02ee9?sharedUserId=mhabir102 · Part 3: How Many RAG Users Fit on One GPU? https://medium.com/@mhabir102/how-many-rag-users-fit-on-one-gpu-a-five-step-check-4a347ee00019?sharedUserId=mhabir102 KV Cache Size: How Many Users Fit on One GPU? https://pub.towardsai.net/kv-cache-size-how-many-users-fit-on-one-gpu-76bcd3c02ee9 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.