{"slug": "how-many-rag-users-fit-on-one-gpu-a-five-step-check", "title": "How Many RAG Users Fit on One GPU? A Five-Step Check", "summary": "A five-step capacity check published in Towards AI shows that raising top-k retrieved chunks from 3 to 20 cuts how many concurrent RAG users one GPU can serve by roughly 70% across three open models: Llama 3 8B falls from 133 to 38 users, Qwen3-30B-A3B from 46 to 13, and gpt-oss-20b from 710 to 212. The analysis, which builds on Sebastian Raschka's Big LLM Architecture Comparison and the series' earlier KV cache formula, notes that Qwen3-30B-A3B's 30.5B parameters take about 57 GiB in 16-bit even though only 3.3B activate per token, so it does not fit while Llama 3 8B does, and that storing Qwen3's cache in 8-bit brings the total to 74 GiB. The author estimates a reranker passing 5 good chunks instead of 20 mediocre ones yields about 2.6 to 2.8x more users, and cautions the workload is illustrative rather than a benchmark of any real system.", "body_md": "You spent weeks on retrieval quality. Chunk sizes, embeddings, a reranker, evaluation sets. Then you picked the answer-generating model the way most teams do: by leaderboard score and parameter count.\n\nA month later you are in a meeting about why the GPU bill is double the estimate, or why latency spikes at 3 p.m. every day.\n\nIn RAG apps, memory is often the first limit you hit, and you can estimate it in ten minutes before you benchmark anything.\n\n**TL;DR**\n\nThis is Part 3 of a series. [Part 2](LINK) derives the KV cache formula and gives you a script; [Part 1](LINK) is the non-technical version. The architecture comparisons behind it are Sebastian Raschka’s [Big LLM Architecture Comparison](https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison).\n\nA chatbot prompt might be 200 tokens. A RAG prompt is your system instructions, the user’s question, conversation history and every retrieved chunk. Eight chunks of 500 tokens is already 4,000 tokens before anything else.\n\nEvery one of those tokens sits in the KV cache for as long as the request is alive, and every concurrent user has their own copy. So in RAG the cache is often what limits how many users one GPU can serve at once, while the long prefill drives latency.\n\nThis is a made-up workload to show the method, not a benchmark of any real system.\n\nI compared three open models that could plausibly run on one GPU. Their per-token cache sizes come from the formula in [Part 2](LINK).\n\nThe table makes four points.\n\nThe smallest cache is not the smallest bill. Qwen3–30B-A3B has a smaller cache than Llama 3 8B (4 KV heads against 8, even with more layers), yet it is the one that does not fit. Its 30.5B parameters take about 57 GiB in 16-bit.\n\nActive parameters are not memory. Qwen3–30B-A3B activates only 3.3B parameters per token, which makes it fast, but you still have to load all 30.5B. The “A3B” saves compute, not memory.\n\nOne lever can flip the answer. Storing Qwen3’s cache in 8-bit brings the total to 74 GiB and it fits. That lever needs a quality test first.\n\nAttention design changes the picture. gpt-oss-20b ships 4-bit expert weights (21B total parameters, 3.6B active) and alternates sliding-window layers with full-attention layers, so its cache stays small. That saving depends on your inference engine actually freeing out-of-window cache, so confirm it on yours.\n\nFitting is only the first filter. Whether a model answers your questions well enough is a separate test (step 5 below).\n\nTop-k is usually tuned for answer quality, and nobody looks at what it does to capacity.\n\nGoing from 3 to 20 retrieved chunks, capacity falls from 133 to 38 users for Llama 3 8B, 46 to 13 for Qwen3–30B-A3B, and 710 to 212 for gpt-oss-20b. Each model’s budget is whatever memory is left after weights and overhead, which is why Qwen3’s line sits lowest. Every model loses roughly 70% of its users.\n\nIf a reranker lets you pass 5 good chunks instead of 20 mediocre ones, you improve quality and capacity together: about 2.6 to 2.8x more users in this example. That is often the cheapest change available, and you can test it in an afternoon.\n\nPost your top-k, P95 prompt length, model and GPU in a response. I’ll run the fit check on a few of them and write up the results.\n\n**LLM Architecture for People Who Ship:** Part 1: [Why One AI Model Can Serve 4x More Users on the Same GPU](https://medium.com/@mhabir102/why-one-ai-model-can-serve-4x-more-users-on-the-same-gpu-06d6387a9559?sharedUserId=mhabir102) · Part 2: [KV Cache Size: How Many Users Fit on One GPU?](https://medium.com/@mhabir102/kv-cache-size-how-many-users-fit-on-one-gpu-76bcd3c02ee9?sharedUserId=mhabir102) · Part 3: [How Many RAG Users Fit on One GPU?](LINK)\n\n[How Many RAG Users Fit on One GPU? A Five-Step Check](https://pub.towardsai.net/how-many-rag-users-fit-on-one-gpu-a-five-step-check-4a347ee00019) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/how-many-rag-users-fit-on-one-gpu-a-five-step-check", "canonical_source": "https://pub.towardsai.net/how-many-rag-users-fit-on-one-gpu-a-five-step-check-4a347ee00019?source=rss----98111c9905da---4", "published_at": "2026-11-21 05:47:35+00:00", "updated_at": "2026-11-21 06:12:48.043565+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools"], "entities": ["Qwen3-30B-A3B", "Llama 3 8B", "gpt-oss-20b", "Sebastian Raschka", "Towards AI", "Medium"], "also_reported_by": [], "takeaway": "Going from 3 to 20 retrieved chunks cuts concurrent users by roughly 70% (Llama 3 8B: 133 to 38; Qwen3-30B-A3B: 46 to 13; gpt-oss-20b: 710 to 212), while a reranker passing 5 good chunks instead of 20 yields about 2.6 to 2.8x more users.", "recommendation": {"advice": "Size KV cache from top-k times chunk length before picking a model, and use a reranker to pass 5 strong chunks instead of 20 mediocre ones for roughly 2.6 to 2.8x more concurrent users.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/how-many-rag-users-fit-on-one-gpu-a-five-step-check", "markdown": "https://wpnews.pro/news/how-many-rag-users-fit-on-one-gpu-a-five-step-check.md", "text": "https://wpnews.pro/news/how-many-rag-users-fit-on-one-gpu-a-five-step-check.txt", "jsonld": "https://wpnews.pro/news/how-many-rag-users-fit-on-one-gpu-a-five-step-check.jsonld"}}