{"slug": "measuring-the-real-concurrency-ceiling-of-an-llm-agent-runner", "title": "Measuring the real concurrency ceiling of an LLM agent runner", "summary": "Andréas, a full-stack developer and CTO, benchmarked his local AI agent runner and found that the concurrency bottleneck was not the model server or hardware but the scheduling policy. The runner's event loop handled requests instantly, while Ollama's model inference and background memory extraction caused wall-clock time to scale to 4.80× the baseline at N=4, worse than serial. He also discovered that coding jobs were hardcoded to run one per repository, and replaced that with a strict path-overlap predicate to safely increase concurrency.", "body_md": "I wanted to raise the concurrency limits on my local AI agent runner. The UI now supports multiple terminal panes running in flight, and my gut told me the runner process itself was becoming the bottleneck.\n\nBefore touching a single config setting, I wrote a benchmark to test that assumption: **When N sessions run at once, what actually breaks — the model server, the hardware, or the scheduling policy?**\n\nIt turns out my gut was entirely wrong.\n\nI wrote a bench script that fires N concurrent chat sessions against the runner. Each turn hits a local model (`hermes3`\n\nvia Ollama) with a fixed prompt, running through the full pipeline: pre-turn intent classification, chat response, and post-reply memory extraction.\n\nI tracked four specific metrics:\n\n| N | Mean ttfa | Mean first-Ollama | Done (Min / Mean / Max) | Wall (Done) | Wall (Drained) | CPU Max | Min Free RAM |\n|---|---|---|---|---|---|---|---|\n1 |\n0.0s | 0.9s | 2.1s / 2.1s / 2.1s | 2.1s | 9.5s | 23% | 6,004 MB |\n2 |\n0.0s | 1.1s | 3.5s / 4.2s / 5.0s | 5.0s | 12.9s | 37% | 5,957 MB |\n4 |\n0.0s | 1.5s | 5.9s / 7.7s / 10.3s | 10.3s | 20.7s | 63% | 5,889 MB |\n\nAt N=4, the wall-clock time scaled to **4.80×** the single-session baseline. Because 4.8× exceeds 4×, concurrent requests actually performed *worse* than a pure, perfectly ordered serial queue.\n\nThe bottleneck breakdown was immediately clear:\n\n`ttfa`\n\nstayed at 0.0s across all runs. The runner's event loop processed and routed incoming POST requests in milliseconds without queuing.`hermes3 8.0B Q4_0`\n\n). Furthermore, `wall (drained)`\n\nproved that background memory extraction keeps Ollama saturated long after the user gets their answer — a hidden tax naive RPS benchmarks completely miss.\n\nChat is only half the system. The real heavy lifting happens in coding jobs, which talk to the Claude API instead of Ollama.\n\nWhen I checked why coding jobs weren't running concurrently, I didn't need a load test. I just needed to look at the scheduler code:\n\n``` js\nconst MAX_CONCURRENT = Number(process.env.MAX_CONCURRENT_JOBS ?? 2)\n\n// In the scheduler pump:\nif (repoHasRunningJob(job.repo)) continue;   // One job per repo, unconditionally\n```\n\nAlmost every ticket in my queue targets the same primary repository. Regardless of what `MAX_CONCURRENT`\n\nwas set to, **the system was hardcoded to run one job per repo at a time.**\n\nThe rule exists for a valid reason: two AI agents branching off the same moving `HEAD`\n\ncreate messy merge conflicts. But serializing the entire repository by default was a blunt policy constraint, not a system capability limit.\n\nTo fix job concurrency without causing merge chaos, I replaced the blind \"one job per repo\" check with a strict path-overlap predicate.\n\nTwo queued jobs (A and B) in the same repo can now run concurrently **only if all three conditions are met**:\n\n`git diff`\n\nfile list rather than their written prose description.If there's any ambiguity, the scheduler defaults to serial execution. A false positive (running serially when safe) costs a few minutes of queue time; a false negative (running concurrently and corrupting state) costs an hour of untangling merge conflicts by hand.\n\nMeasure first. The limit you think you're hitting is rarely the one actually holding you back.\n\n*I'm Andréas — full-stack dev, CTO at a B2B SaaS, building my own agent tooling. Portfolio: https://andreas-bodin.vercel.app*", "url": "https://wpnews.pro/news/measuring-the-real-concurrency-ceiling-of-an-llm-agent-runner", "canonical_source": "https://dev.to/arti0/measuring-the-real-concurrency-ceiling-of-an-llm-agent-runner-53cb", "published_at": "2026-08-15 14:24:21+00:00", "updated_at": "2026-08-15 14:42:46.006620+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "machine-learning"], "entities": ["Andréas", "Ollama", "hermes3", "Claude API"], "alternates": {"html": "https://wpnews.pro/news/measuring-the-real-concurrency-ceiling-of-an-llm-agent-runner", "markdown": "https://wpnews.pro/news/measuring-the-real-concurrency-ceiling-of-an-llm-agent-runner.md", "text": "https://wpnews.pro/news/measuring-the-real-concurrency-ceiling-of-an-llm-agent-runner.txt", "jsonld": "https://wpnews.pro/news/measuring-the-real-concurrency-ceiling-of-an-llm-agent-runner.jsonld"}}