Testing LLM Concurrency on Consumer Hardware #
I recently watched a YouTube video of someone testing a server-grade LLM hardware setup, pushing it to see just how much concurrency it could actually handle. It got me thinking: what can your own — perhaps a bit above-average — “gaming” / “workstation” PC really do? Especially within the limits of my RTX 5060 and its fast but limited 8GB of VRAM. I’ve been thinking about building a game or simulation driven by a high agent count, and I wanted to know what the feasible limit really is. That question brought me to these tests — and the results you’ll find below: the rig, the method, and every number and chart for all 16 models, individually and side by side. A note on authorship: that opening paragraph is my own words — the rest of this post is mostly AI-generated, though I have carefully gone over all of it, and every number has been checked against the raw benchmark data before publishing. And the dataset isn’t done yet — I plan to keep testing more models and will add them to the list as I do, so keep a lookout for new entries in the sidebar.
TL;DR #
Concurrency scales on consumer hardware — pooled agents multiplied throughput
8.7x at best. The top all-in result was
MiniCPM5 1B at 983 tok/s
(433% of its solo speed).
The worst was a flatline: Qwen3.5 0.8B with MTP enabled never scaled at all
(~180 tok/s at any concurrency) because multi-token prediction serializes requests —
with MTP off the same model hit 787 tok/s.
Thinking models hide a huge share of their budget in reasoning_content
(up to 100%), and time-to-first-token degrades from milliseconds to tens of seconds under load. Errors across all 384 runs: zero.
Test rig #
One desktop, fully loaded while every run executed. This is a dirty daily-driver environment — not a clean lab bench — and that was the point: the results are what real consumer hardware actually does.
Ryzen 9 9950X3D
Zen 5 V-Cache flagship. Core affinity is hard-locked to CCD0 so every LLM loop stays on the 3D V-Cache cores — zero cross-CCD thread-hopping on the Infinity Fabric.
32GB DDR5 6400
UCLK locked 1:1 with the memory controller for the lowest possible system↔GPU transfer latency during CPU-side MoE offload and high-concurrency prefill pooling.
RTX 5060 8GB GDDR7
Core + VRAM overclocked to squeeze maximum raw bandwidth out of the 128-bit bus. Every model was fully offloaded — the GPU is the engine room.
3 monitors, always on
2x ultrawide + 1x 4K primary, with a hardware-accelerated multi-tab Chrome profile running through every single benchmark run.
LM Studio configuration #
Identical server settings for every model — differences (MTP, KV cache quant, context caps) are called out per model on its page.
⚒ Decode & scheduling
- GPU offloadMax for model
- CPU threads8 · CCD0 - Eval batch size2048
- Physical batch size512
- Max concurrency24
- Flash attention On
● Memory & context
- Context length34304 · max - Unified KV cache On - Offload KV to GPU On - Keep model in memory On - Try mmap() On - KV cache quantF16 · Q8_0 · Q4_0
⚡ Decoding strategy
- Speculative decoding Off - MTP exceptionQwen3.5 0.8B
- KV quant rulesF16 ≤1.5B · Q8_0 2B · Q4_0 3B+
- Context cap32k or 8k · per model
Methodology #
The harness is a zero-dependency Node agent loop (benchmark/benchmark.js
) plus a
sweep driver (benchmark/sweep.js
) streaming against LM Studio’s local OpenAI-compatible
endpoint. Each model got 24 sequential runs —
one per concurrency level 1–24 — of 60 seconds wall time. Each agent holds a
12,000-token budget, runs a multi-turn loop until budget or timeout, and samples
temperature 0.9
.
Token-level timing captures TTFT (time to first token), per-stream token gaps,
content vs reasoning tokens, and per-agent/combined throughput. Reasoning is counted differently
per model family: reasoning_content
streams (MiniCPM5, Qwen-DeepSeek, Chronos, Qwen3.5),
inline `<think>...</think>`
tags stripped from content (Gemma Heretic), or not at all
(the clean non-thinking models). Per-model pages note which.
One honesty flag: some per-agent numbers carry *
— agents that sat queued for ~11–16s and then generated in a short burst report inflated per-agent tok/s. Those are timing artifacts; combined throughput and TTFT are the trusted columns.
Leaderboard — side by side #
| # | Model | Solo tok/s | Peak tok/s | @ agents | Scaling | TTFT solo | TTFT @ peak | Content @24 | Reasoning @24 | Errors |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | ||||||||||
Qwen2.5 0.5BQwen · 0.5B · KV F16Falcon3 1B InstructFalcon · 1B · KV F16Qwen-DeepSeek 1.5B Agentic DistillQwen · 1.5B · KV F16 · thinkingQwen3.5 0.8B — MTP DisabledQwen · 0.8B · KV F16 · thinkingGemma3 1B IT (Heretic thinking)Gemma · 1B · KV F16 · thinkingLlama 3.2 1B Mini-AgentLlama · 1B · KV F16Qwen2.5 1.5BQwen · 1.5B · KV F16Gemma 2B IT (smashed)Gemma · 2B · KV Q8_0Chronos 1.5BChronos · 1.5B · KV F16 · thinkingUnsloth Ministral 3 3B (2512)Ministral · 3B · KV Q4_0Mistralai Ministral 3 3B (2512)Ministral · 3B · KV Q4_0Llama 3.2 3B InstructLlama · 3B · KV Q4_0Granite 4.1 3BIBM Granite · 3B · KV Q4_0Qwen3 4BQwen · 4B · KV Q4_0 · thinkingQwen3.5 0.8B — MTP EnabledQwen · 0.8B · KV F16 · thinkingPodium rows mark the top 3 by peak combined throughput. Solo = concurrency 1 · Peak = highest combined all-token tok/s across 1–24 · Scaling = peak as % of solo · Content/Reasoning @24 = token mix at 24 agents · Every model page has the full 24-row table and raw data.
Side-by-side charts #
All charts below are generated from the sweep JSON in the site’s dark theme; original matplotlib exports are downloadable per model.
Models — individually #
Every model gets its own page: the complete 24-row table, key findings, all 8–9 charts, and raw CSV/JSON downloads.
[01 · MiniCPM · 1B 983 tok/s · 433% scaling
](models/minicpm5-1b-thinking.php)
MiniCPM5 1B
All-in throughput king (983 tok/s) but half its output is hidden reasoning and TTFT climbs to 52s.
[02 · Qwen · 0.5B 944 tok/s · 267% scaling
](models/qwen2.5-0.5b.php)
Qwen2.5 0.5B
The throughput monster — fastest solo of all (354 tok/s) and fastest visible-content rate on record (944 tok/s at 20 agents) on full-precision F16 KV.
[03 · Falcon · 1B 839 tok/s · 449% scaling
](models/falcon3-1b.php)
Falcon3 1B Instruct
Fastest pure-content scorer — 839 tok/s at 13 agents, then rolls off as the 8k context saturates.
[04 · Qwen · 1.5B 829 tok/s · 502% scaling
](models/qwen-deepseek-1.5b.php)
Qwen-DeepSeek 1.5B Agentic Distill
Best scaling efficiency (5.0x) but the worst TTFT profile — 20–58s to first token under load.
[05 · Qwen · 0.8B 787 tok/s · 329% scaling
](models/qwen3.5-0.8b-mtp-off.php)
Qwen3.5 0.8B — MTP Disabled
The MTP experiment: with multi-token prediction off, throughput scales 3.3x instead of flatlining at ~180 tok/s.
[06 · Gemma · 1B 719 tok/s · 420% scaling
](models/gemma-3-1b-heretic.php)
### Gemma3 1B IT (Heretic thinking)
Friendliest thinking profile — 86ms solo TTFT, 719 tok/s all-in, and it just thinks harder under load.
[07 · Llama · 1B 652 tok/s · 333% scaling
](models/llama-3.2-1b-mini-agent.php)
Llama 3.2 1B Mini-Agent
The first agentic-tuned model tested — clean 196 tok/s solo, plateau of ~600–650 tok/s from 13 agents on.
[08 · Qwen · 1.5B 577 tok/s · 334% scaling
](models/qwen2.5-1.5b.php)
Qwen2.5 1.5B
Best latency at low concurrency (108ms TTFT) and 3.3x scaling — but an erratic mid-range TTFT curve.
[09 · Gemma · 2B 538 tok/s · 445% scaling
](models/gemma-2b-smashed.php)
### Gemma 2B IT (smashed)
Middle of the pack and steady — 538 tok/s peak, and the best latency of the smaller models (61ms–2.4s).
[10 · Chronos · 1.5B 458 tok/s · 520% scaling
](models/chronos-1.5b.php)
Chronos 1.5B
Slowest solo (88 tok/s) but scales 5.2x — 94% of its output at high concurrency is hidden reasoning.
[11 · Ministral · 3B 410 tok/s · 872% scaling
](models/unsloth-ministral-3-3b.php)
### Unsloth Ministral 3 3B (2512)
Slowest solo of any model tested (47 tok/s on Q4_0 KV) but the highest relative scaling on record — 8.7x to 410 tok/s at 23 agents.
[12 · Ministral · 3B 404 tok/s · 569% scaling
](models/mistralai-ministral-3-3b.php)
### Mistralai Ministral 3 3B (2512)
The official build vs the unsloth sibling at the same Q4_0 KV: 71 tok/s solo, smooth monotonic scaling to 404 tok/s at 22 agents — faster at every concurrency below the top end.
[13 · Llama · 3B 404 tok/s · 425% scaling
](models/llama-3.2-3b.php)
Llama 3.2 3B Instruct
The 3B anchor — only 95 tok/s solo, but the smoothest, most predictable scaling curve in the sweep (4.25x).
[14 · IBM Granite · 3B 362 tok/s · 458% scaling
](models/granite-4.1-3b.php)
Granite 4.1 3B
Slowest solo of all (79 tok/s) and the noisiest scaling curve, but TTFT is the real weak point — 20–24s at 20+ agents.
[15 · Qwen · 4B 340 tok/s · 567% scaling
](models/qwen3-4b.php)
Qwen3 4B
The worst latency profile of any model tested — 11.8s TTFT solo, 40–56s under load — and ~95% of its output is hidden reasoning at high concurrency. Peak: 340 tok/s.
[16 · Qwen · 0.8B 202 tok/s · 100% scaling
](models/qwen3.5-0.8b-mtp.php)
Qwen3.5 0.8B — MTP Enabled
The cautionary tale: MTP serializes requests — flat ~180 tok/s at any concurrency and zero visible content from 10 agents up.
Key takeaways #
Concurrency pays, up to a wall. Every model except MTP-on Qwen3.5 scaled 2.7–8.7x — the one outlier at the low end being Qwen2.5 0.5B, which barely needs concurrency (527 tok/s at just 2 agents). Diminishing returns beyond ~13–16 agents for most; a few (MiniCPM5, Falcon3, Qwen2.5 0.5B) kept climbing to 20+.MTP serializes — it is not a throughput win for concurrency. The cleanest result of the whole sweep: same model, same box, MTP off = 3.3x scaling; MTP on = flatline. The decode path cannot batch.Thinking tokens are overhead, not content. At 24 agents, reasoning share ran 45–100%. If your agents need visible answers, count content-only throughput.TTFT is the agent-latency wall. Prefill-heavy thinking models push first-token latency to 20–60s under load — Qwen3 4B waits up to the full 60s window — unusable for interactive orchestration, fine for batch pipelines.Reliability is not the bottleneck. 288 runs, zero hard errors. Timeouts (budget exhaustion) are the only non-OK outcome.A 9950X3D + 32GB + 8GB VRAM can host the whole swarm. 24 concurrent 1B–3B agents ran with KV on GPU and no OOMs — this is genuinely consumer-hardware territory.
What’s next — bigger models & real-world use #
This sweep is a living dataset — it’s not done yet. I plan to keep testing and adding models to the list as they land, so keep a lookout for new entries in the sidebar. Next up on the bench:
Bigger models— climbing to roughly** 9B parameters**, about the max I can fit in 8GB of VRAM, to see how the concurrency curve changes as models outgrow the card.** MoE models with offloaded experts**— sparse models that push expert weights out to system RAM; the 9950X3D’s CCD0 affinity and 32GB of 1:1 DDR5 were built for exactly this. The question: does expert off serialize under load, or does MoE’s sparsity actually hold up better under concurrency than dense models?
And throughput only answers how fast a model can go — not how well it works. The strongest performers from this sweep also head onto real-world use datasets under the same harness and the same dirty rig:
Agentic task mixes— multi-turn tool calling, structured JSON/action output, and code-editing loops, measuring parse success and instruction-following while 8–24 agents are in flight.Long-context drift— whether models stay coherent when agents hold full 32k contexts under compaction pressure, not just 60-second sprints.** Quality vs token budget**— the reasoning-share question made concrete: does the hidden thinking actually buy better task outcomes, or just slower answers?Latency budgets— which models stay inside interactive TTFT budgets at realistic concurrency, and which are batch-only.
Data & downloads #
Every number above is reproducible: each model page links its sweep-summary.csv
,
sweep-summary.json
, the full report markdown, and the original white-background matplotlib
chart exports. The generation pipeline lives in build/
(dark-theme chart generator + model page builder) and is re-runnable as the dataset grows.