Measuring the real concurrency ceiling of an LLM agent runner

Andréas, a full-stack developer and CTO, benchmarked his local AI agent runner and found that the concurrency bottleneck was not the model server or hardware but the scheduling policy. The runner's event loop handled requests instantly, while Ollama's model inference and background memory extraction caused wall-clock time to scale to 4.80× the baseline at N=4, worse than serial. He also discovered that coding jobs were hardcoded to run one per repository, and replaced that with a strict path-overlap predicate to safely increase concurrency.

I wanted to raise the concurrency limits on my local AI agent runner. The UI now supports multiple terminal panes running in flight, and my gut told me the runner process itself was becoming the bottleneck. Before touching a single config setting, I wrote a benchmark to test that assumption: When N sessions run at once, what actually breaks — the model server, the hardware, or the scheduling policy? It turns out my gut was entirely wrong. I wrote a bench script that fires N concurrent chat sessions against the runner. Each turn hits a local model hermes3 via Ollama with a fixed prompt, running through the full pipeline: pre-turn intent classification, chat response, and post-reply memory extraction. I tracked four specific metrics: | N | Mean ttfa | Mean first-Ollama | Done Min / Mean / Max | Wall Done | Wall Drained | CPU Max | Min Free RAM | |---|---|---|---|---|---|---|---| 1 | 0.0s | 0.9s | 2.1s / 2.1s / 2.1s | 2.1s | 9.5s | 23% | 6,004 MB | 2 | 0.0s | 1.1s | 3.5s / 4.2s / 5.0s | 5.0s | 12.9s | 37% | 5,957 MB | 4 | 0.0s | 1.5s | 5.9s / 7.7s / 10.3s | 10.3s | 20.7s | 63% | 5,889 MB | At N=4, the wall-clock time scaled to 4.80× the single-session baseline. Because 4.8× exceeds 4×, concurrent requests actually performed worse than a pure, perfectly ordered serial queue. The bottleneck breakdown was immediately clear: ttfa stayed at 0.0s across all runs. The runner's event loop processed and routed incoming POST requests in milliseconds without queuing. hermes3 8.0B Q4 0 . Furthermore, wall drained proved that background memory extraction keeps Ollama saturated long after the user gets their answer — a hidden tax naive RPS benchmarks completely miss. Chat is only half the system. The real heavy lifting happens in coding jobs, which talk to the Claude API instead of Ollama. When I checked why coding jobs weren't running concurrently, I didn't need a load test. I just needed to look at the scheduler code: js const MAX CONCURRENT = Number process.env.MAX CONCURRENT JOBS ?? 2 // In the scheduler pump: if repoHasRunningJob job.repo continue; // One job per repo, unconditionally Almost every ticket in my queue targets the same primary repository. Regardless of what MAX CONCURRENT was set to, the system was hardcoded to run one job per repo at a time. The rule exists for a valid reason: two AI agents branching off the same moving HEAD create messy merge conflicts. But serializing the entire repository by default was a blunt policy constraint, not a system capability limit. To fix job concurrency without causing merge chaos, I replaced the blind "one job per repo" check with a strict path-overlap predicate. Two queued jobs A and B in the same repo can now run concurrently only if all three conditions are met : git diff file list rather than their written prose description.If there's any ambiguity, the scheduler defaults to serial execution. A false positive running serially when safe costs a few minutes of queue time; a false negative running concurrently and corrupting state costs an hour of untangling merge conflicts by hand. Measure first. The limit you think you're hitting is rarely the one actually holding you back. I'm Andréas — full-stack dev, CTO at a B2B SaaS, building my own agent tooling. Portfolio: https://andreas-bodin.vercel.app