I wanted to raise the concurrency limits on my local AI agent runner. The UI now supports multiple terminal panes running in flight, and my gut told me the runner process itself was becoming the bottleneck.
Before touching a single config setting, I wrote a benchmark to test that assumption: When N sessions run at once, what actually breaks — the model server, the hardware, or the scheduling policy?
It turns out my gut was entirely wrong.
I wrote a bench script that fires N concurrent chat sessions against the runner. Each turn hits a local model (hermes3
via Ollama) with a fixed prompt, running through the full pipeline: pre-turn intent classification, chat response, and post-reply memory extraction.
I tracked four specific metrics:
| N | Mean ttfa | Mean first-Ollama | Done (Min / Mean / Max) | Wall (Done) | Wall (Drained) | CPU Max | Min Free RAM |
|---|---|---|---|---|---|---|---|
| 1 | |||||||
| 0.0s | 0.9s | 2.1s / 2.1s / 2.1s | 2.1s | 9.5s | 23% | 6,004 MB | |
| 2 | |||||||
| 0.0s | 1.1s | 3.5s / 4.2s / 5.0s | 5.0s | 12.9s | 37% | 5,957 MB | |
| 4 | |||||||
| 0.0s | 1.5s | 5.9s / 7.7s / 10.3s | 10.3s | 20.7s | 63% | 5,889 MB |
At N=4, the wall-clock time scaled to 4.80× the single-session baseline. Because 4.8× exceeds 4×, concurrent requests actually performed worse than a pure, perfectly ordered serial queue.
The bottleneck breakdown was immediately clear:
ttfa
stayed at 0.0s across all runs. The runner's event loop processed and routed incoming POST requests in milliseconds without queuing.hermes3 8.0B Q4_0
). Furthermore, wall (drained)
proved that background memory extraction keeps Ollama saturated long after the user gets their answer — a hidden tax naive RPS benchmarks completely miss.
Chat is only half the system. The real heavy lifting happens in coding jobs, which talk to the Claude API instead of Ollama.
When I checked why coding jobs weren't running concurrently, I didn't need a load test. I just needed to look at the scheduler code:
const MAX_CONCURRENT = Number(process.env.MAX_CONCURRENT_JOBS ?? 2)
// In the scheduler pump:
if (repoHasRunningJob(job.repo)) continue; // One job per repo, unconditionally
Almost every ticket in my queue targets the same primary repository. Regardless of what MAX_CONCURRENT
was set to, the system was hardcoded to run one job per repo at a time.
The rule exists for a valid reason: two AI agents branching off the same moving HEAD
create messy merge conflicts. But serializing the entire repository by default was a blunt policy constraint, not a system capability limit.
To fix job concurrency without causing merge chaos, I replaced the blind "one job per repo" check with a strict path-overlap predicate.
Two queued jobs (A and B) in the same repo can now run concurrently only if all three conditions are met:
git diff
file list rather than their written prose description.If there's any ambiguity, the scheduler defaults to serial execution. A false positive (running serially when safe) costs a few minutes of queue time; a false negative (running concurrently and corrupting state) costs an hour of untangling merge conflicts by hand.
Measure first. The limit you think you're hitting is rarely the one actually holding you back.
I'm Andréas — full-stack dev, CTO at a B2B SaaS, building my own agent tooling. Portfolio: https://andreas-bodin.vercel.app