How Many RAG Users Fit on One GPU? A Five-Step Check A five-step capacity check published in Towards AI shows that raising top-k retrieved chunks from 3 to 20 cuts how many concurrent RAG users one GPU can serve by roughly 70% across three open models: Llama 3 8B falls from 133 to 38 users, Qwen3-30B-A3B from 46 to 13, and gpt-oss-20b from 710 to 212. The analysis, which builds on Sebastian Raschka's Big LLM Architecture Comparison and the series' earlier KV cache formula, notes that Qwen3-30B-A3B's 30.5B parameters take about 57 GiB in 16-bit even though only 3.3B activate per token, so it does not fit while Llama 3 8B does, and that storing Qwen3's cache in 8-bit brings the total to 74 GiB. The author estimates a reranker passing 5 good chunks instead of 20 mediocre ones yields about 2.6 to 2.8x more users, and cautions the workload is illustrative rather than a benchmark of any real system. You spent weeks on retrieval quality. Chunk sizes, embeddings, a reranker, evaluation sets. Then you picked the answer-generating model the way most teams do: by leaderboard score and parameter count. A month later you are in a meeting about why the GPU bill is double the estimate, or why latency spikes at 3 p.m. every day. In RAG apps, memory is often the first limit you hit, and you can estimate it in ten minutes before you benchmark anything. TL;DR This is Part 3 of a series. Part 2 LINK derives the KV cache formula and gives you a script; Part 1 LINK is the non-technical version. The architecture comparisons behind it are Sebastian Raschka’s Big LLM Architecture Comparison https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison . A chatbot prompt might be 200 tokens. A RAG prompt is your system instructions, the user’s question, conversation history and every retrieved chunk. Eight chunks of 500 tokens is already 4,000 tokens before anything else. Every one of those tokens sits in the KV cache for as long as the request is alive, and every concurrent user has their own copy. So in RAG the cache is often what limits how many users one GPU can serve at once, while the long prefill drives latency. This is a made-up workload to show the method, not a benchmark of any real system. I compared three open models that could plausibly run on one GPU. Their per-token cache sizes come from the formula in Part 2 LINK . The table makes four points. The smallest cache is not the smallest bill. Qwen3–30B-A3B has a smaller cache than Llama 3 8B 4 KV heads against 8, even with more layers , yet it is the one that does not fit. Its 30.5B parameters take about 57 GiB in 16-bit. Active parameters are not memory. Qwen3–30B-A3B activates only 3.3B parameters per token, which makes it fast, but you still have to load all 30.5B. The “A3B” saves compute, not memory. One lever can flip the answer. Storing Qwen3’s cache in 8-bit brings the total to 74 GiB and it fits. That lever needs a quality test first. Attention design changes the picture. gpt-oss-20b ships 4-bit expert weights 21B total parameters, 3.6B active and alternates sliding-window layers with full-attention layers, so its cache stays small. That saving depends on your inference engine actually freeing out-of-window cache, so confirm it on yours. Fitting is only the first filter. Whether a model answers your questions well enough is a separate test step 5 below . Top-k is usually tuned for answer quality, and nobody looks at what it does to capacity. Going from 3 to 20 retrieved chunks, capacity falls from 133 to 38 users for Llama 3 8B, 46 to 13 for Qwen3–30B-A3B, and 710 to 212 for gpt-oss-20b. Each model’s budget is whatever memory is left after weights and overhead, which is why Qwen3’s line sits lowest. Every model loses roughly 70% of its users. If a reranker lets you pass 5 good chunks instead of 20 mediocre ones, you improve quality and capacity together: about 2.6 to 2.8x more users in this example. That is often the cheapest change available, and you can test it in an afternoon. Post your top-k, P95 prompt length, model and GPU in a response. I’ll run the fit check on a few of them and write up the results. LLM Architecture for People Who Ship: Part 1: Why One AI Model Can Serve 4x More Users on the Same GPU https://medium.com/@mhabir102/why-one-ai-model-can-serve-4x-more-users-on-the-same-gpu-06d6387a9559?sharedUserId=mhabir102 · Part 2: KV Cache Size: How Many Users Fit on One GPU? https://medium.com/@mhabir102/kv-cache-size-how-many-users-fit-on-one-gpu-76bcd3c02ee9?sharedUserId=mhabir102 · Part 3: How Many RAG Users Fit on One GPU? LINK How Many RAG Users Fit on One GPU? A Five-Step Check https://pub.towardsai.net/how-many-rag-users-fit-on-one-gpu-a-five-step-check-4a347ee00019 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.