# Why One AI Model Can Serve 4x More Users on the Same GPU

> Source: <https://pub.towardsai.net/why-one-ai-model-can-serve-4x-more-users-on-the-same-gpu-06d6387a9559?source=rss----98111c9905da---4>
> Published: 2026-11-21 05:04:03+00:00

**Recommendation:** Compute KV cache size per conversation from the model's public config before choosing a model, and compare models on users served per dollar of GPU memory, not leaderboard score alone.
More in the playbook "Inference and self-hosting": <https://wpnews.pro/playbooks/inference-serving>

Your team picked an AI model because it topped a leaderboard. A month after launch, the GPU bill arrives and nobody can explain it.

The leaderboard didn’t lie. It just never measured the thing that costs you money.

Most model comparisons stop at how smart a model is. They skip how much memory it needs to *remember a conversation*. That second number decides how many customers one machine can serve at the same time.

Sebastian Raschka’s [Big LLM Architecture Comparison](https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison) lines up the major open models side by side. What stands out is how similar the overall structure is, from GPT-2 in 2019 to today’s flagships.

So where do the big differences come from? From a few small design choices inside the model. They don’t show up in a benchmark. They show up in your invoice.

Imagine a restaurant where each waiter must remember every table’s whole conversation, word for word, for the entire evening.

AI models work the same way. The notepad is called the **KV cache**. It’s the memory the model keeps so it doesn’t have to re-read the whole conversation for every new word. Newer models use shorthand tricks, so the same machine can hold far more conversations at once.

Here is what that looks like in numbers, worked out from the published configs of two real models from the same family. Set aside about 40 GiB of GPU memory for conversations, each about 3,000 words long. Llama 2 7B, with the older full-attention design, fits about **20** at once. Llama 3 8B, a model of similar size with the newer grouped-query design, fits about **80**. Same hardware, 4x the capacity.Other designs go further, and some fit over a hundred. The full math and a script to check any model are in Part 2 of this series.

You don’t need the math. You need five questions to ask your team or vendor before choosing a model:

Treat memory per conversation as a first-class metric next to accuracy and latency. It is computable from the model’s public configuration file, in minutes, before you benchmark anything. That is Part 2, KV Cache Size: How Many Users Fit on One GPU?, with a script you can run.

Don’t just ask “which model is best?” Ask “which model is best *per dollar of GPU memory* for my workload?”

Those are different questions, and the second one is the one your finance team will ask in a month.

Which of the five questions could your team answer today? Reply with the number.

[Why One AI Model Can Serve 4x More Users on the Same GPU](https://pub.towardsai.net/why-one-ai-model-can-serve-4x-more-users-on-the-same-gpu-06d6387a9559) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.
