# KV Cache Size: How Many Users Fit on One GPU?

> Source: <https://pub.towardsai.net/kv-cache-size-how-many-users-fit-on-one-gpu-76bcd3c02ee9?source=rss----98111c9905da---4>
> Published: 2026-11-21 05:47:33+00:00

**Recommendation:** Size GPU memory for concurrent users by computing per-token KV cache from config.json before deployment; in 40 GiB, Llama 3 8B fits 80 4k-token conversations and DeepSeek V3 fits 149.
More in the playbook "Inference and self-hosting": <https://wpnews.pro/playbooks/inference-serving>

DeepSeek V3 has 671 billion parameters. Its KV cache per token is roughly half the size of Llama 3 8B’s. That cache decides how many users fit on your GPU, and you can compute it from a config file in two minutes.

Sebastian Raschka’s [Big LLM Architecture Comparison](https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison) explains the design choices behind this: grouped-query attention, latent attention, sliding windows. I wanted to know what they cost in bytes, because that is the number on your GPU bill.

**TL;DR**

When an LLM writes a reply, each new token has to look back at every token before it. Recomputing all of that at every step would be far too slow, so the model keeps the intermediate results (the *keys* and *values*) in GPU memory and reuses them.

That saves compute. It costs memory. The cache grows with every token, and every user in the batch has their own copy.

So two things you control, context length and number of concurrent users, matter as much as the model’s parameter count.

For standard attention (multi-head or grouped-query):

```
bytes per token = 2 × layers × kv_heads × head_dim × bytes per value
```

The 2 is for keys and values. Everything else comes from the model’s config.json.

For Llama 3 8B with a 16-bit cache, that is 131,072 bytes, or 128 KiB per token. A 4,000-token conversation holds about 0.5 GiB of cache. A hundred of them in parallel is about 49 GiB, before you have loaded a single weight.

I ran the formula on five public models, using values from each model’s config or paper. The cache is 16-bit. The last column assumes you reserve 40 GiB of GPU memory for cache and every user has a 4k-token conversation, which every model here supports natively.

† Beyond the model’s native context, so a what-if number. * gpt-oss sliding-window layers stop growing after 128 tokens, so past that point each extra token costs about 24 KiB. That only holds if your inference engine actually frees out-of-window cache. Check before you rely on it.

In the same 40 GiB, the full-attention design fits 20 conversations and Llama 3 8B fits 80. DeepSeek V3, roughly 80 to 100 times larger by parameter count, fits 149.

**Grouped-query attention (GQA): share keys and values.** Llama 2 7B has 32 attention heads, and each keeps its own keys and values. Llama 3 8B keeps 32 query heads but only 8 sets of keys and values, so the cache falls from 512 to 128 KiB per token, exactly 4x.

**Multi-head latent attention (MLA): compress them.** DeepSeek V3 does not store full keys and values. It stores one compressed vector per layer (512 latent values plus 64 positional ones) and rebuilds what it needs. That is 61 × 576 × 2 bytes, about 68.6 KiB per token, which is about half of Llama 3 8B’s cache.

**Sliding-window attention: stop remembering everything.** Some layers only look at the most recent tokens. gpt-oss-20b alternates sliding-window layers (128 tokens) with full-attention layers, so half its layers can keep a fixed-size cache however long the conversation gets, if the engine frees the rest. Raschka covers the same idea in Gemma 3, which mixes local and global layers at a 5:1 ratio.

The gap widens as conversations get longer:

Say these out loud if you reuse the numbers:

The core of the script is this:

``` python
def per_token_bytes(layers, kv_heads, head_dim, bytes_per_value=2):    return 2 * layers * kv_heads * head_dim * bytes_per_value
print(per_token_bytes(32, 32, 128) / 1024)  # Llama 2 7B: 512.0 KiB per tokenprint(per_token_bytes(32, 8, 128) / 1024)   # Llama 3 8B: 128.0 KiB per token
```

The full version, with MLA and sliding-window handling, is here: [*\[GitHub link\]*](https://github.com/Mehedi-Hasan-Abir/kv-cache-calculator.git). Copy a model’s config.json values into the config list and it prints the table for you.

What does the number come out to for the model you run? Post it in the comments, and say which model and context length.

Next: [How Many RAG Users Fit on One GPU?](LINK), using this math as the first filter.

**LLM Architecture for People Who Ship:** Part 1: [Why One AI Model Can Serve 4x More Users on the Same GPU](https://medium.com/@mhabir102/why-one-ai-model-can-serve-4x-more-users-on-the-same-gpu-06d6387a9559?sharedUserId=mhabir102) · Part 2: [KV Cache Size: How Many Users Fit on One GPU?](https://medium.com/@mhabir102/kv-cache-size-how-many-users-fit-on-one-gpu-76bcd3c02ee9?sharedUserId=mhabir102) · Part 3: [How Many RAG Users Fit on One GPU?](https://medium.com/@mhabir102/how-many-rag-users-fit-on-one-gpu-a-five-step-check-4a347ee00019?sharedUserId=mhabir102)

[KV Cache Size: How Many Users Fit on One GPU?](https://pub.towardsai.net/kv-cache-size-how-many-users-fit-on-one-gpu-76bcd3c02ee9) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.
