Why One AI Model Can Serve 4x More Users on the Same GPU Llama 3 8B can serve about 80 concurrent 3,000-word conversations in roughly 40 GiB of GPU memory, versus about 20 for Llama 2 7B on the same hardware — a 4x capacity gain, according to an analysis citing Sebastian Raschka's Big LLM Architecture Comparison. The difference comes from grouped-query attention, which shrinks the KV cache each conversation requires, a cost driver that standard benchmarks do not measure. The analysis argues teams should treat memory per conversation as a first-class metric alongside accuracy and latency, computable from a model's public configuration file before benchmarking. Your team picked an AI model because it topped a leaderboard. A month after launch, the GPU bill arrives and nobody can explain it. The leaderboard didn’t lie. It just never measured the thing that costs you money. Most model comparisons stop at how smart a model is. They skip how much memory it needs to remember a conversation . That second number decides how many customers one machine can serve at the same time. Sebastian Raschka’s Big LLM Architecture Comparison https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison lines up the major open models side by side. What stands out is how similar the overall structure is, from GPT-2 in 2019 to today’s flagships. So where do the big differences come from? From a few small design choices inside the model. They don’t show up in a benchmark. They show up in your invoice. Imagine a restaurant where each waiter must remember every table’s whole conversation, word for word, for the entire evening. AI models work the same way. The notepad is called the KV cache . It’s the memory the model keeps so it doesn’t have to re-read the whole conversation for every new word. Newer models use shorthand tricks, so the same machine can hold far more conversations at once. Here is what that looks like in numbers, worked out from the published configs of two real models from the same family. Set aside about 40 GiB of GPU memory for conversations, each about 3,000 words long. Llama 2 7B, with the older full-attention design, fits about 20 at once. Llama 3 8B, a model of similar size with the newer grouped-query design, fits about 80 . Same hardware, 4x the capacity.Other designs go further, and some fit over a hundred. The full math and a script to check any model are in Part 2 of this series. You don’t need the math. You need five questions to ask your team or vendor before choosing a model: Treat memory per conversation as a first-class metric next to accuracy and latency. It is computable from the model’s public configuration file, in minutes, before you benchmark anything. That is Part 2, KV Cache Size: How Many Users Fit on One GPU?, with a script you can run. Don’t just ask “which model is best?” Ask “which model is best per dollar of GPU memory for my workload?” Those are different questions, and the second one is the one your finance team will ask in a month. Which of the five questions could your team answer today? Reply with the number. Why One AI Model Can Serve 4x More Users on the Same GPU https://pub.towardsai.net/why-one-ai-model-can-serve-4x-more-users-on-the-same-gpu-06d6387a9559 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.