cd /news/ai-agents/throwing-headcount-at-the-problem · home › topics › ai-agents › article
[ARTICLE · art-172680] src=frontier.replit.com ↗ pub= topic=ai-agents verified=true sentiment=↑ positive

Throwing Headcount at the Problem

Replit reported that a swarm of 32 Claude Opus 5.5 agents made NumPy workloads 18.9× faster in two hours, versus 3.7× for a single Opus 5.5 agent given the same two hours, in performance-engineering tests the company ran to study inference-time compute scaling and coordination. Replit said GPT-6.2 Astra was the strongest solo agent tested, but its swarms got worse as they grew, and that the best-performing swarms had workers claim tasks and share results on a common board rather than routing everything through a coordinator. Replit is putting the approach into users' hands through Missions, which launches a swarm of agents toward a user-supplied goal, and argues existing benchmarks like ProgramBench understate teamwork because their work is additive and their scores are capped.

recommendation

Organize agent swarms around a shared task board where workers claim work and post results, and avoid funneling all messages through a coordinator whose inbox grows with each added agent.

more in playbook: Designing agent systems
by read14 min views10 publishedNov 19, 2026
Throwing Headcount at the Problem
Image: Frontier (auto-discovered)

Fred Brooks observed in 1975 that adding programmers to a late project makes it later because every new engineer brings more coordination than work. Early research on agent swarms found that multi-agent systems often trailed a single agent once coordination costs were accounted for [1–4]. We tested whether this still holds for today’s models using performance engineering tasks to examine how scaling inference-time compute drives both speedups and emergent coordination mechanics. On NumPy, one Claude Opus 5.5 agent made the workloads 3.7× faster in two hours, while a swarm of 32 Opus agents, given the same two hours, reached 18.9×. GPT-6.2 Astra was the strongest solo agent we tested, yet its swarms got worse as they grew. Given no instructions on how to organize, each model’s swarms arranged themselves differently, and the differences mattered. The best swarms sent few direct messages; workers claimed tasks and shared results on a common board. The ones that struggled routed everything through the coordinator, whose inbox grew with every agent added. The other deciding factor was the shape of the work. While codebases with independent problems allowed each new agent to contribute usefully, highly constrained tasks often led extra agents to compete for the same code.

Agent Habitat #

On Replit, an agent swarm works much like an engineering team. Each agent, coordinator or worker, is a full Replit Agent with its own cloud sandbox, so it can build, run, and profile code without getting in anyone else’s way. Agents share a repository rather than a filesystem, and their work comes together through branches and pushes. They also share context through three communication channels: a swarm board that lets agents open threads, reply, address posts to specific teammates, and set their status; direct messaging between teammates; and transcripts that let agents read what a teammate tried and why without interrupting it.

Agent Habitat provides the elastic compute behind this. Each sandbox is an isolated Linux microVM whose CPU, memory, and disk grow with the workload, so a coordinator can start hundreds of workers quickly, and a single agent can work across several sandboxes at once. Network access, credentials, and runtime activity are locked down and traced per agent, and execution state is journaled durably so a swarm survives failures.

Putting Swarms to Work #

With models that can coordinate and a harness built for teams, swarms are starting to be practical for real work. Missions puts this in users’ hands. Give Replit a goal, and it launches a swarm of agents to pursue it. The research question is no longer whether swarms can work, but rather when a team beats one agent, which models make good teammates, and how a swarm should organize itself to get there.

Current benchmarks don’t fully answer these questions. Most published measurements of agent teams use build tasks such as ProgramBench [5, 6], where agents reconstruct a program and are scored on how many tests pass. Two properties of these tasks limit what they can show about teamwork. The work is additive, so pieces built in parallel combine without agents needing to learn from one another. The score is also capped, because a single agent already passes nearly every test, so a team can finish sooner but can’t score higher, and on small problems coordination overhead erases even that.

Hard problems usually don’t look like this. In open research, and in most real engineering, progress comes from search and experimentation, which means trying ideas that may fail, learning from what others found, and combining partial results that don’t simply add up. A useful test of teamwork needs that shape, and a score that rewards finding more, not just finishing earlier.

Performance engineering has both the search and the open-ended score, plus a fast, objective feedback loop. An agent can profile a workload, change a hot path, and learn within a reasonable time whether it ran faster without changing the output. The score has no ceiling, since there is no fixed limit on how much faster the code can get. Agents can split up the work, but they can’t ignore each other. Patches to shared code conflict at merge, and agents that don’t share results repeat each other’s failed attempts. Coordination shows up directly in the score. A team that works well together can find more than any one agent, and a team that doesn’t can do worse.

Optimization-Bench #

We are releasing Optimization-Bench, a suite of performance-optimization tasks evaluated on pinned versions of popular open-source libraries. Each task provides an agent with the library’s source code, driver code with a realistic workload, a strict output contract, and an isolated verifier. The goal is to improve the wall-clock time for processing the workload under a strict set of constraints.

We selected libraries where profiling showed most of each workload’s runtime falling inside the library’s own code rather than in external dependencies. That way, the hot paths sit in files agents are allowed to edit, leaving meaningful scope to optimize. Each selected library has more than one workload (a task it can handle), and these workloads often share common infrastructure. For example, in NumPy, there are myriad functions, such as calculating quantiles, sorting arrays, and filtering for conditions. We consider each distinct function a single workload. For each library selected, the agent is prompted to optimize a subset of all workloads that target core, high-traffic functionality with stable long-term specifications.

Building on this approach, this initial prerelease focuses on 14 widely used open-source libraries under active maintenance, demonstrating that agent swarms on Replit can still achieve significant performance gains. We discuss the results and analysis of four of the largest ones:

  • NumPy has 13 workloads, including quantiles, sorting, datetime handling, and random numbers.
  • SQLite has 12, including full-text search, indexing, JSON, and joins.
  • SQLGlot has 4, all running through one parser and generator.
  • Git has 7, all confined to a 15-file diff library.

Task Setup

Every agent was allocated 8 vCPUs, 16 GiB of RAM, a two-hour execution window, and standard profiling tools. Agents must preserve all observable outputs, error handling, and public interfaces. They were given rules that strictly prohibit cross-call caching, special-casing inputs, result precomputation, delegating tasks to external libraries, or altering process-wide runtime settings (e.g., allocators or garbage collection).

Each benchmark task was evaluated across four configurations: a single agent and swarms of 8, 16, and 32 agents. We tested six frontier models: Claude Opus 5.5, Claude Sonnet 5.5, GPT-6.2 Astra, GPT-6 Astra, GPT-6.1 Sol, and GPT-6 Luna (all set to medium reasoning effort, except Luna at high reasoning effort).

The coordinator receives the task requirements but no instructions on how to organize the team. How the swarm divides the work and integrates branches is left entirely to the agents. Agents coordinate via direct messaging and a shared swarm board. Each agent operates inside an isolated sandbox VM, sharing only the repository context. Network access is disabled, and all runs are audited post-execution to guarantee compliance. Only the final submission branch at the two-hour mark is evaluated.

Task Evaluation

Studies have shown that cloud VMs are unreliable for measuring performance, because speed varies between instances and over time [7, 8]. Even on dedicated hardware, environment size, link order, and memory layout can shift results enough to reverse a conclusion [9, 10]. Interleaved, repeated trials are the standard fix [11, 12].

Agents are prone to reward hacking and exploiting flaws in the scoring rather than solving the task [13, 14]. METR caught models overwriting the timer and reading the grader’s answer from the call stack [13], and Sakana’s reported 10–100× CUDA speedups came from a flaw in its evaluation sandbox [15].

In Optimization-Bench, at the two-hour deadline, each run’s final branch is frozen and sent to a separate VM that the agents can never access. The evaluation pipeline rejects edits outside the permitted files and rebuilds the original and candidate from source. A candidate must pass the repo’s tests and match the original’s output on public and hidden inputs. It must also pass generated tests that only the pipeline holds, which catch submissions that overfit to the existing tests or exploit gaps in them. Timing runs only on the hidden inputs, on a dedicated Intel machine,<sup>1</sup> one evaluation at a time. The original and candidate runs alternate in randomized order, so neither benefits from warm caches or drift over the session. The timer and the correctness checks run outside the candidate’s process, so the code being measured can’t reach the clock or the checker.

Results #

Median speedup on NumPy’s 13 workloads with 1, 8, 16, and 32 agents, where the dashed line marks no speedup.

Median speedup on SQLite’s 12 workloads with 1, 8, 16, and 32 agents, where the dashed line marks no speedup.

Median speedup on SQLGlot’s 4 transpilation workloads with 1, 8, 16, and 32 agents, where the dashed line marks no speedup.

Median speedup on Git’s 7 diff workloads with 1, 8, 16, and 32 agents, where the dashed line marks no speedup.

We found that Anthropic models improved performance when working in a swarm, and the magnitude of the speedups correlated with how much independent work each codebase offered. For instance, NumPy had 13 separate workloads and yielded the largest jump, from 3.66× to 18.92×. Even in repositories like SQLite and SQLGlot, where solo agents struggled to find gains, swarms still performed well enough. Git, although it had 7 workloads, introduced the most overlap, since all workloads shared the same 15 files, and Anthropic swarms struggled to outperform their solo runs. In contrast, OpenAI models did not see similar correlations. Their swarms beat their solo runs in some repos and fell behind in others, and on SQLGlot none of them meaningfully improved on their solo runs. GPT-6.2 Astra had the strongest solo runs on every task, but its swarms peaked at 8 agents and, by 32, fell below its solo score on NumPy, SQLGlot, and Git.

One example is NumPy’s searchsorted function, which finds where each value belongs in a sorted array. On main, it runs the binary search across all keys together, doing the first comparison for every key, then the second for every key, and so on. With millions of keys, that data is too large for the CPU cache, so every step reloads it from memory. The Opus swarm changed it to search 256 keys at a time, a batch small enough to stay in cache from start to finish. It also skips the slower NaN-safe comparison when neither array contains NaN. On float64 inputs with many keys, searchsorted became 8.7–13.1× faster with identical results.

Analysis #

A model’s solo score was a poor predictor of how its swarm performed. The topology each swarm settled into mattered more. Opus 5.5 and Sonnet 5.5 swarms beat their solo runs every time, by a median of 1.30× at 16 and 32 agents, while GPT-6 Astra and GPT-6.1 Sol swarms mostly did no better than working alone. GPT-6.2 Astra paints this picture most clearly. It was the strongest solo agent we tested, more than three times faster than Opus alone on NumPy, yet its swarms got worse as they grew, ending below its solo score on three of four tasks at 32 agents. Opus moved the other way, gaining more as the team grew.

The swarms that scaled kept coordination in durable places any agent could read later. Opus 5.5, for instance, sent 82% of its messages to the board, mostly in per-workload threads where workers claimed pieces and reported findings. GPT-6 Astra and GPT-6.1 Sol did the opposite. Their workers sent, on average, about 22 and 20 direct messages each to the coordinator, against 1.4 for Opus workers. GPT-6.2 Astra fell into the same habit, with fewer than 40% of its messages on the board and around 15 direct messages per worker to the coordinator. The over-reliance on direct messaging moved valuable information off the board and let it fall out of date.

For the swarms that worked best, the coordinator handed out work, waited, and merged what came back. Opus coordinators sent about 0.5 direct messages per worker, usually only the final stop message, and in the 32-agent NumPy run the coordinator never read the board or messaged a worker at all. GPT-6 Astra and GPT-6.1 Sol coordinators instead acted as switchboards, answering a constant stream of messages. Luna coordinators went further. One polled the board more than 200 times, another did much of the work itself instead of delegating it, and some told their workers to stay idle. GPT-6.2 Astra’s coordinators showed a third pattern. Workers sent them hundreds of direct messages per run, but they rarely replied or posted to the board, so little of what workers reported reached the rest of the team. Swarms performed best when each worker owned a distinct workload, file, or code area and said so publicly. Opus swarms opened a thread per NumPy workload, posted file claims like “touching btree.c” before editing SQLite, and negotiated ownership between named pairs of workers. Even minimal coordination works under this structure. For instance, we found that Sonnet 5.5 communicated the least of any model, yet its swarms beat its solo runs in every case.

Doubling the team from 16 to 32 agents separated the swarms clearly. For Opus, direct messages per agent fell as the team grew, from 4.9 to 1.0 on NumPy and from 3.1 to 1.1 on Git. For GPT-6 Astra and GPT-6.1 Sol, every added worker added load to one coordinator. Sol’s coordinator went from 356 to 631 messages received on SQLGlot, and GPT-6 Astra’s from 346 to 615 on NumPy. The coordinator became the bottleneck, and doubling its load barely moved scores. GPT-6.2 Astra hit the same wall. When its team doubled, the coordinator’s inbox grew on all four tasks, about fourfold on SQLite, and its scores fell on all four. Its strongest swarms were mostly its 8-agent teams, where the coordinator’s inbox was smallest.

The best solo agents did not make the best teams. GPT-6.2 Astra had the strongest solo runs on NumPy, SQLite, SQLGlot, and Git, yet Opus swarms scored higher on NumPy, SQLite, and SQLGlot. Git was the exception because its workloads all share the same 15 files, leaving little for a large team to divide, and GPT-6.2 Astra matched or beat Opus at every team size. This matches what others have found. Cursor’s swarms improved when planners stopped implementing and workers stopped planning [16], and Cognition reports multi-agent patterns working when many agents contribute but writes stay single-threaded [17]. Our results add a caution: models differ a great deal in how they organize on their own, and that difference decides whether the extra agents are worth it.

Next Steps #

This post covers four of the libraries in Optimization-Bench. We’re adding more in the open-source repository, across kinds of software where the hot paths look very different: compression (bzip2, xz), code formatters and linters (Black, SQLFluff), JSON processing (jq, serde_json), an analytical database (DuckDB), media processing (FFmpeg), and search indexing (Tantivy).

Acknowledgments #

We thank all members of the Replit AI Research team, Agent Infrastructure team, and Core Platform team for building out the primitives for Replit’s swarm infrastructure and Agent Habitat.

References #

[1] [Khatua, Zhu et al., CooperBench: Why Coding Agents Cannot be Your Teammates Yet](https://arxiv.org/abs/2601.13295)

[2] [Cemri et al., Why Do Multi-Agent LLM Systems Fail?](https://arxiv.org/abs/2503.13657)

[3] [Kim et al., Towards a Science of Scaling Agent Systems](https://arxiv.org/abs/2512.08296)

[4] [Cognition, Don’t Build Multi-Agents](https://cognition.com/blog/dont-build-multi-agents)

[5] Anthropic, Claude Opus 5.5 System Card, §8.12

[6] [Anthropic, Claude Fable 5.2 and Claude Mythos 5.2 System Card](https://assets.claude.com/27d60918d218125e7504bc7354c4b4626ec56b11.pdf)

[8] [Laaber et al., Software Microbenchmarking in the Cloud. How Bad Is It Really?](https://doi.org/10.1007/s10664-019-09681-1)

[9] [Mytkowicz et al., Producing Wrong Data Without Doing Anything Obviously Wrong!](https://doi.org/10.1145/1508244.1508275)

[10] [Curtsinger & Berger, STABILIZER: Statistically Sound Performance Evaluation](https://doi.org/10.1145/2451116.2451141)

[11] Abedi & Brecht, Conducting Repeatable Experiments in Highly Variable Cloud Computing Environments

[12] [Kalibera & Jones, Rigorous Benchmarking in Reasonable Time](https://doi.org/10.1145/2464157.2464160)

[13] [METR, Recent Frontier Models Are Reward Hacking](https://metr.org/blog/2025-06-05-recent-reward-hacking/)

[14] [Zhong et al., ImpossibleBench: Measuring LLMs’ Propensity of Exploiting Test Cases](https://arxiv.org/abs/2510.20270)

[15] TechCrunch, Sakana walks back claims that its AI can dramatically speed up model training

[16] [Cursor, Agent swarms and the new model economics](https://cursor.com/blog/agent-swarm-model-economics)

[17] [Cognition, Multi-Agents: What’s Actually Working](https://cognition.com/blog/multi-agents-working)

Footnotes #

The timing machine is a Google Cloud c4-standard-32 instance in us-central1-a with an Intel Xeon Platinum 8581C @ 2.30 GHz (AVX-512 and AMX). Hyperthreading is off, so it has 16 physical cores on one socket and one memory node, with 260 MiB of L3 cache. Benchmarks run on cores 0–7; the operating system and Docker use cores 8–15. It runs Ubuntu 24.04 (kernel 7.0.0-1014-gcp) and Docker 29.9 on a 300 GB hyperdisk-balanced disk. The VM has no cloud credentials and no GitHub access; submissions are copied in by hand and checksum-checked.↩

── more in #ai-agents 4 stories · sorted by recency
── more on @replit 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/throwing-headcount-a…] indexed:0 read:14min 2026-11-19 · —