{"slug": "testing-llm-concurrency-on-consumer-hardware-rtx-5060", "title": "Testing LLM Concurrency on Consumer Hardware (RTX 5060)", "summary": "A benchmark test by an unnamed developer found that LLM concurrency scales on consumer hardware, with pooled agents multiplying throughput by up to 8.7x, and the top result being MiniCPM5 1B at 983 tok/s (433% of its solo speed). The worst performer was Qwen3.5 0.8B with multi-token prediction enabled, which never scaled beyond ~180 tok/s, while the same model without MTP hit 787 tok/s. Across 384 runs on an RTX 5060 8GB system, there were zero errors, but time-to-first-token degraded from milliseconds to tens of seconds under load.", "body_md": "## Testing LLM Concurrency on Consumer Hardware\n\nI recently watched a YouTube video of someone testing a server-grade LLM hardware setup, pushing it to see just how much concurrency it could actually handle. It got me thinking: what can your own — perhaps a bit above-average — “gaming” / “workstation” PC really do? Especially within the limits of my RTX 5060 and its fast but limited 8GB of VRAM. I’ve been thinking about building a game or simulation driven by a high agent count, and I wanted to know what the feasible limit really is. That question brought me to these tests — and the results you’ll find below: the rig, the method, and every number and chart for all 16 models, individually and side by side. A note on authorship: that opening paragraph is my own words — the rest of this post is mostly AI-generated, though I have carefully gone over all of it, and every number has been checked against the raw benchmark data before publishing. And the dataset isn’t done yet — I plan to keep testing more models and will add them to the list as I do, so keep a lookout for new entries in the sidebar.\n\n## TL;DR\n\nConcurrency **scales** on consumer hardware — pooled agents multiplied throughput\n8.7x at best. The top all-in result was\nMiniCPM5 1B at **983 tok/s**\n(433% of its solo speed).\nThe worst was a flatline: **Qwen3.5 0.8B with MTP enabled** never scaled at all\n(~180 tok/s at any concurrency) because multi-token prediction serializes requests —\nwith MTP off the same model hit 787 tok/s.\nThinking models hide a huge share of their budget in `reasoning_content`\n\n(up to 100%),\nand time-to-first-token degrades from milliseconds to tens of seconds under load. Errors across all\n384 runs: **zero**.\n\n## Test rig\n\nOne desktop, fully loaded while every run executed. This is a *dirty* daily-driver environment — not a clean lab bench — and that was the point: the results are what real consumer hardware actually does.\n\n### Ryzen 9 9950X3D\n\nZen 5 V-Cache flagship. Core affinity is hard-locked to **CCD0** so every LLM loop stays on the 3D V-Cache cores — zero cross-CCD thread-hopping on the Infinity Fabric.\n\n### 32GB DDR5 6400\n\nUCLK locked 1:1 with the memory controller for the lowest possible system↔GPU transfer latency during CPU-side MoE offload and high-concurrency prefill pooling.\n\n### RTX 5060 8GB GDDR7\n\nCore + VRAM overclocked to squeeze maximum raw bandwidth out of the 128-bit bus. Every model was fully offloaded — the GPU is the engine room.\n\n### 3 monitors, always on\n\n2x ultrawide + 1x 4K primary, with a hardware-accelerated multi-tab Chrome profile running through every single benchmark run.\n\n## LM Studio configuration\n\nIdentical server settings for every model — differences (MTP, KV cache quant, context caps) are called out per model on its page.\n\n### ⚒ Decode & scheduling\n\n- GPU offloadMax for model\n- CPU threads8\n*· CCD0* - Eval batch size2048\n- Physical batch size512\n- Max concurrency24\n- Flash attention\n*On*\n\n### ● Memory & context\n\n- Context length34304\n*· max* - Unified KV cache\n*On* - Offload KV to GPU\n*On* - Keep model in memory\n*On* - Try mmap()\n*On* - KV cache quantF16 · Q8_0 · Q4_0\n\n### ⚡ Decoding strategy\n\n- Speculative decoding\n*Off* - MTP exceptionQwen3.5 0.8B\n- KV quant rulesF16 ≤1.5B · Q8_0 2B · Q4_0 3B+\n- Context cap32k or 8k · per model\n\n## Methodology\n\nThe harness is a zero-dependency Node agent loop (`benchmark/benchmark.js`\n\n) plus a\nsweep driver (`benchmark/sweep.js`\n\n) streaming against LM Studio’s local OpenAI-compatible\nendpoint. Each model got **24 sequential runs** —\none per concurrency level 1–24 — of **60 seconds wall time**. Each agent holds a\n**12,000-token budget**, runs a multi-turn loop until budget or timeout, and samples\n`temperature 0.9`\n\n.\n\nToken-level timing captures **TTFT** (time to first token), per-stream token gaps,\ncontent vs reasoning tokens, and per-agent/combined throughput. Reasoning is counted differently\nper model family: `reasoning_content`\n\nstreams (MiniCPM5, Qwen-DeepSeek, Chronos, Qwen3.5),\ninline `<think>...</think>`\n\ntags stripped from content (Gemma Heretic), or not at all\n(the clean non-thinking models). Per-model pages note which.\n\nOne honesty flag: some per-agent numbers carry `*`\n\n— agents that sat queued for\n~11–16s and then generated in a short burst report inflated per-agent tok/s. Those are\ntiming artifacts; **combined throughput and TTFT are the trusted columns**.\n\n## Leaderboard — side by side\n\n| # | Model | Solo tok/s | Peak tok/s | @ agents | Scaling | TTFT solo | TTFT @ peak | Content @24 | Reasoning @24 | Errors |\n|---|---|---|---|---|---|---|---|---|---|---|\n| 1 |\n|\n\n[Qwen2.5 0.5B](models/qwen2.5-0.5b.php)Qwen · 0.5B · KV F16[Falcon3 1B Instruct](models/falcon3-1b.php)Falcon · 1B · KV F16[Qwen-DeepSeek 1.5B Agentic Distill](models/qwen-deepseek-1.5b.php)Qwen · 1.5B · KV F16 · thinking[Qwen3.5 0.8B — MTP Disabled](models/qwen3.5-0.8b-mtp-off.php)Qwen · 0.8B · KV F16 · thinking[Gemma3 1B IT (Heretic thinking)](models/gemma-3-1b-heretic.php)Gemma · 1B · KV F16 · thinking[Llama 3.2 1B Mini-Agent](models/llama-3.2-1b-mini-agent.php)Llama · 1B · KV F16[Qwen2.5 1.5B](models/qwen2.5-1.5b.php)Qwen · 1.5B · KV F16[Gemma 2B IT (smashed)](models/gemma-2b-smashed.php)Gemma · 2B · KV Q8_0[Chronos 1.5B](models/chronos-1.5b.php)Chronos · 1.5B · KV F16 · thinking[Unsloth Ministral 3 3B (2512)](models/unsloth-ministral-3-3b.php)Ministral · 3B · KV Q4_0[Mistralai Ministral 3 3B (2512)](models/mistralai-ministral-3-3b.php)Ministral · 3B · KV Q4_0[Llama 3.2 3B Instruct](models/llama-3.2-3b.php)Llama · 3B · KV Q4_0[Granite 4.1 3B](models/granite-4.1-3b.php)IBM Granite · 3B · KV Q4_0[Qwen3 4B](models/qwen3-4b.php)Qwen · 4B · KV Q4_0 · thinking[Qwen3.5 0.8B — MTP Enabled](models/qwen3.5-0.8b-mtp.php)Qwen · 0.8B · KV F16 · thinkingPodium rows mark the top 3 by peak combined throughput. Solo = concurrency 1 · Peak = highest combined all-token tok/s across 1–24 · Scaling = peak as % of solo · Content/Reasoning @24 = token mix at 24 agents · Every model page has the full 24-row table and raw data.\n\n## Side-by-side charts\n\nAll charts below are generated from the sweep JSON in the site’s dark theme; original matplotlib exports are downloadable per model.\n\n## Models — individually\n\nEvery model gets its own page: the complete 24-row table, key findings, all 8–9 charts, and raw CSV/JSON downloads.\n\n[01 · MiniCPM · 1B\n983 tok/s · 433% scaling\n](models/minicpm5-1b-thinking.php)\n\n### MiniCPM5 1B\n\nAll-in throughput king (983 tok/s) but half its output is hidden reasoning and TTFT climbs to 52s.\n\n[02 · Qwen · 0.5B\n944 tok/s · 267% scaling\n](models/qwen2.5-0.5b.php)\n\n### Qwen2.5 0.5B\n\nThe throughput monster — fastest solo of all (354 tok/s) and fastest visible-content rate on record (944 tok/s at 20 agents) on full-precision F16 KV.\n\n[03 · Falcon · 1B\n839 tok/s · 449% scaling\n](models/falcon3-1b.php)\n\n### Falcon3 1B Instruct\n\nFastest pure-content scorer — 839 tok/s at 13 agents, then rolls off as the 8k context saturates.\n\n[04 · Qwen · 1.5B\n829 tok/s · 502% scaling\n](models/qwen-deepseek-1.5b.php)\n\n### Qwen-DeepSeek 1.5B Agentic Distill\n\nBest scaling efficiency (5.0x) but the worst TTFT profile — 20–58s to first token under load.\n\n[05 · Qwen · 0.8B\n787 tok/s · 329% scaling\n](models/qwen3.5-0.8b-mtp-off.php)\n\n### Qwen3.5 0.8B — MTP Disabled\n\nThe MTP experiment: with multi-token prediction off, throughput scales 3.3x instead of flatlining at ~180 tok/s.\n\n[06 · Gemma · 1B\n719 tok/s · 420% scaling\n](models/gemma-3-1b-heretic.php)\n\n### Gemma3 1B IT (Heretic thinking)\n\nFriendliest thinking profile — 86ms solo TTFT, 719 tok/s all-in, and it just thinks harder under load.\n\n[07 · Llama · 1B\n652 tok/s · 333% scaling\n](models/llama-3.2-1b-mini-agent.php)\n\n### Llama 3.2 1B Mini-Agent\n\nThe first agentic-tuned model tested — clean 196 tok/s solo, plateau of ~600–650 tok/s from 13 agents on.\n\n[08 · Qwen · 1.5B\n577 tok/s · 334% scaling\n](models/qwen2.5-1.5b.php)\n\n### Qwen2.5 1.5B\n\nBest latency at low concurrency (108ms TTFT) and 3.3x scaling — but an erratic mid-range TTFT curve.\n\n[09 · Gemma · 2B\n538 tok/s · 445% scaling\n](models/gemma-2b-smashed.php)\n\n### Gemma 2B IT (smashed)\n\nMiddle of the pack and steady — 538 tok/s peak, and the best latency of the smaller models (61ms–2.4s).\n\n[10 · Chronos · 1.5B\n458 tok/s · 520% scaling\n](models/chronos-1.5b.php)\n\n### Chronos 1.5B\n\nSlowest solo (88 tok/s) but scales 5.2x — 94% of its output at high concurrency is hidden reasoning.\n\n[11 · Ministral · 3B\n410 tok/s · 872% scaling\n](models/unsloth-ministral-3-3b.php)\n\n### Unsloth Ministral 3 3B (2512)\n\nSlowest solo of any model tested (47 tok/s on Q4_0 KV) but the highest relative scaling on record — 8.7x to 410 tok/s at 23 agents.\n\n[12 · Ministral · 3B\n404 tok/s · 569% scaling\n](models/mistralai-ministral-3-3b.php)\n\n### Mistralai Ministral 3 3B (2512)\n\nThe official build vs the unsloth sibling at the same Q4_0 KV: 71 tok/s solo, smooth monotonic scaling to 404 tok/s at 22 agents — faster at every concurrency below the top end.\n\n[13 · Llama · 3B\n404 tok/s · 425% scaling\n](models/llama-3.2-3b.php)\n\n### Llama 3.2 3B Instruct\n\nThe 3B anchor — only 95 tok/s solo, but the smoothest, most predictable scaling curve in the sweep (4.25x).\n\n[14 · IBM Granite · 3B\n362 tok/s · 458% scaling\n](models/granite-4.1-3b.php)\n\n### Granite 4.1 3B\n\nSlowest solo of all (79 tok/s) and the noisiest scaling curve, but TTFT is the real weak point — 20–24s at 20+ agents.\n\n[15 · Qwen · 4B\n340 tok/s · 567% scaling\n](models/qwen3-4b.php)\n\n### Qwen3 4B\n\nThe worst latency profile of any model tested — 11.8s TTFT solo, 40–56s under load — and ~95% of its output is hidden reasoning at high concurrency. Peak: 340 tok/s.\n\n[16 · Qwen · 0.8B\n202 tok/s · 100% scaling\n](models/qwen3.5-0.8b-mtp.php)\n\n### Qwen3.5 0.8B — MTP Enabled\n\nThe cautionary tale: MTP serializes requests — flat ~180 tok/s at any concurrency and zero visible content from 10 agents up.\n\n## Key takeaways\n\n**Concurrency pays, up to a wall.** Every model except MTP-on Qwen3.5 scaled 2.7–8.7x — the one outlier at the low end being Qwen2.5 0.5B, which barely needs concurrency (527 tok/s at just 2 agents). Diminishing returns beyond ~13–16 agents for most; a few (MiniCPM5, Falcon3, Qwen2.5 0.5B) kept climbing to 20+.**MTP serializes — it is not a throughput win for concurrency.** The cleanest result of the whole sweep: same model, same box, MTP off = 3.3x scaling; MTP on = flatline. The decode path cannot batch.**Thinking tokens are overhead, not content.** At 24 agents, reasoning share ran 45–100%. If your agents need visible answers, count content-only throughput.**TTFT is the agent-latency wall.** Prefill-heavy thinking models push first-token latency to 20–60s under load — Qwen3 4B waits up to the full 60s window — unusable for interactive orchestration, fine for batch pipelines.**Reliability is not the bottleneck.** 288 runs, zero hard errors. Timeouts (budget exhaustion) are the only non-OK outcome.**A 9950X3D + 32GB + 8GB VRAM can host the whole swarm.** 24 concurrent 1B–3B agents ran with KV on GPU and no OOMs — this is genuinely consumer-hardware territory.\n\n## What’s next — bigger models & real-world use\n\nThis sweep is a **living dataset** — it’s not done yet. I plan to keep\ntesting and adding models to the list as they land, so keep a lookout for new entries in the\nsidebar. Next up on the bench:\n\n**Bigger models**— climbing to roughly** 9B parameters**, about the max I can fit in 8GB of VRAM, to see how the concurrency curve changes as models outgrow the card.** MoE models with offloaded experts**— sparse models that push expert weights out to system RAM; the 9950X3D’s CCD0 affinity and 32GB of 1:1 DDR5 were built for exactly this. The question: does expert offloading serialize under load, or does MoE’s sparsity actually hold up better under concurrency than dense models?\n\nAnd throughput only answers *how fast* a model can go — not *how well it works*.\nThe strongest performers from this sweep also head onto **real-world use datasets**\nunder the same harness and the same dirty rig:\n\n**Agentic task mixes**— multi-turn tool calling, structured JSON/action output, and code-editing loops, measuring parse success and instruction-following while 8–24 agents are in flight.**Long-context drift**— whether models stay coherent when agents hold full 32k contexts under compaction pressure, not just 60-second sprints.** Quality vs token budget**— the reasoning-share question made concrete: does the hidden thinking actually buy better task outcomes, or just slower answers?**Latency budgets**— which models stay inside interactive TTFT budgets at realistic concurrency, and which are batch-only.\n\n## Data & downloads\n\nEvery number above is reproducible: each model page links its `sweep-summary.csv`\n\n,\n`sweep-summary.json`\n\n, the full report markdown, and the original white-background matplotlib\nchart exports. The generation pipeline lives in `build/`\n\n(dark-theme chart generator +\nmodel page builder) and is re-runnable as the dataset grows.", "url": "https://wpnews.pro/news/testing-llm-concurrency-on-consumer-hardware-rtx-5060", "canonical_source": "https://ai.2it.onl/posts/concurrency-sweep/", "published_at": "2026-08-02 06:57:01+00:00", "updated_at": "2026-08-02 07:22:31.017437+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-research"], "entities": ["MiniCPM5", "Qwen3.5", "RTX 5060", "Ryzen 9 9950X3D", "LM Studio"], "alternates": {"html": "https://wpnews.pro/news/testing-llm-concurrency-on-consumer-hardware-rtx-5060", "markdown": "https://wpnews.pro/news/testing-llm-concurrency-on-consumer-hardware-rtx-5060.md", "text": "https://wpnews.pro/news/testing-llm-concurrency-on-consumer-hardware-rtx-5060.txt", "jsonld": "https://wpnews.pro/news/testing-llm-concurrency-on-consumer-hardware-rtx-5060.jsonld"}}