Throwing Headcount at the Problem Replit reported that a swarm of 32 Claude Opus 5.5 agents made NumPy workloads 18.9× faster in two hours, versus 3.7× for a single Opus 5.5 agent given the same two hours, in performance-engineering tests the company ran to study inference-time compute scaling and coordination. Replit said GPT-6.2 Astra was the strongest solo agent tested, but its swarms got worse as they grew, and that the best-performing swarms had workers claim tasks and share results on a common board rather than routing everything through a coordinator. Replit is putting the approach into users' hands through Missions, which launches a swarm of agents toward a user-supplied goal, and argues existing benchmarks like ProgramBench understate teamwork because their work is additive and their scores are capped. Fred Brooks observed in 1975 that adding programmers to a late project makes it later because every new engineer brings more coordination than work. Early research on agent swarms found that multi-agent systems often trailed a single agent once coordination costs were accounted for 1–4 . We tested whether this still holds for today’s models using performance engineering tasks to examine how scaling inference-time compute drives both speedups and emergent coordination mechanics. On NumPy, one Claude Opus 5.5 agent made the workloads 3.7× faster in two hours, while a swarm of 32 Opus agents, given the same two hours, reached 18.9×. GPT-6.2 Astra was the strongest solo agent we tested, yet its swarms got worse as they grew. Given no instructions on how to organize, each model’s swarms arranged themselves differently, and the differences mattered. The best swarms sent few direct messages; workers claimed tasks and shared results on a common board. The ones that struggled routed everything through the coordinator, whose inbox grew with every agent added. The other deciding factor was the shape of the work. While codebases with independent problems allowed each new agent to contribute usefully, highly constrained tasks often led extra agents to compete for the same code. Agent Habitat On Replit, an agent swarm works much like an engineering team. Each agent, coordinator or worker, is a full Replit Agent with its own cloud sandbox, so it can build, run, and profile code without getting in anyone else’s way. Agents share a repository rather than a filesystem, and their work comes together through branches and pushes. They also share context through three communication channels: a swarm board that lets agents open threads, reply, address posts to specific teammates, and set their status; direct messaging between teammates; and transcripts that let agents read what a teammate tried and why without interrupting it. Agent Habitat https://frontier.replit.com/agent-habitat-architecture/ provides the elastic compute behind this. Each sandbox is an isolated Linux microVM whose CPU, memory, and disk grow with the workload, so a coordinator can start hundreds of workers quickly, and a single agent can work across several sandboxes at once. Network access, credentials, and runtime activity are locked down and traced per agent, and execution state is journaled durably so a swarm survives failures. Putting Swarms to Work With models that can coordinate and a harness built for teams, swarms are starting to be practical for real work. Missions https://replit.com/missions puts this in users’ hands. Give Replit a goal, and it launches a swarm of agents to pursue it. The research question is no longer whether swarms can work, but rather when a team beats one agent, which models make good teammates, and how a swarm should organize itself to get there. Current benchmarks don’t fully answer these questions. Most published measurements of agent teams use build tasks such as ProgramBench 5, 6 , where agents reconstruct a program and are scored on how many tests pass. Two properties of these tasks limit what they can show about teamwork. The work is additive, so pieces built in parallel combine without agents needing to learn from one another. The score is also capped, because a single agent already passes nearly every test, so a team can finish sooner but can’t score higher, and on small problems coordination overhead erases even that. Hard problems usually don’t look like this. In open research, and in most real engineering, progress comes from search and experimentation, which means trying ideas that may fail, learning from what others found, and combining partial results that don’t simply add up. A useful test of teamwork needs that shape, and a score that rewards finding more, not just finishing earlier. Performance engineering has both the search and the open-ended score, plus a fast, objective feedback loop. An agent can profile a workload, change a hot path, and learn within a reasonable time whether it ran faster without changing the output. The score has no ceiling, since there is no fixed limit on how much faster the code can get. Agents can split up the work, but they can’t ignore each other. Patches to shared code conflict at merge, and agents that don’t share results repeat each other’s failed attempts. Coordination shows up directly in the score. A team that works well together can find more than any one agent, and a team that doesn’t can do worse. Optimization-Bench We are releasing Optimization-Bench https://github.com/replit/optimization-bench , a suite of performance-optimization tasks evaluated on pinned versions of popular open-source libraries. Each task provides an agent with the library’s source code, driver code with a realistic workload, a strict output contract, and an isolated verifier. The goal is to improve the wall-clock time for processing the workload under a strict set of constraints. We selected libraries where profiling showed most of each workload’s runtime falling inside the library’s own code rather than in external dependencies. That way, the hot paths sit in files agents are allowed to edit, leaving meaningful scope to optimize. Each selected library has more than one workload a task it can handle , and these workloads often share common infrastructure. For example, in NumPy, there are myriad functions, such as calculating quantiles, sorting arrays, and filtering for conditions. We consider each distinct function a single workload. For each library selected, the agent is prompted to optimize a subset of all workloads that target core, high-traffic functionality with stable long-term specifications. Building on this approach, this initial prerelease focuses on 14 widely used open-source libraries under active maintenance, demonstrating that agent swarms on Replit can still achieve significant performance gains. We discuss the results and analysis of four of the largest ones: - NumPy has 13 workloads, including quantiles, sorting, datetime handling, and random numbers. - SQLite has 12, including full-text search, indexing, JSON, and joins. - SQLGlot has 4, all running through one parser and generator. - Git has 7, all confined to a 15-file diff library. Task Setup Every agent was allocated 8 vCPUs, 16 GiB of RAM, a two-hour execution window, and standard profiling tools. Agents must preserve all observable outputs, error handling, and public interfaces. They were given rules that strictly prohibit cross-call caching, special-casing inputs, result precomputation, delegating tasks to external libraries, or altering process-wide runtime settings e.g., allocators or garbage collection . Each benchmark task was evaluated across four configurations: a single agent and swarms of 8, 16, and 32 agents. We tested six frontier models: Claude Opus 5.5, Claude Sonnet 5.5, GPT-6.2 Astra, GPT-6 Astra, GPT-6.1 Sol, and GPT-6 Luna all set to medium reasoning effort, except Luna at high reasoning effort . The coordinator receives the task requirements but no instructions on how to organize the team. How the swarm divides the work and integrates branches is left entirely to the agents. Agents coordinate via direct messaging and a shared swarm board. Each agent operates inside an isolated sandbox VM, sharing only the repository context. Network access is disabled, and all runs are audited post-execution to guarantee compliance. Only the final submission branch at the two-hour mark is evaluated. Task Evaluation Studies have shown that cloud VMs are unreliable for measuring performance, because speed varies between instances and over time 7, 8 . Even on dedicated hardware, environment size, link order, and memory layout can shift results enough to reverse a conclusion 9, 10 . Interleaved, repeated trials are the standard fix 11, 12 . Agents are prone to reward hacking and exploiting flaws in the scoring rather than solving the task 13, 14 . METR caught models overwriting the timer and reading the grader’s answer from the call stack 13 , and Sakana’s reported 10–100× CUDA speedups came from a flaw in its evaluation sandbox 15 . In Optimization-Bench, at the two-hour deadline, each run’s final branch is frozen and sent to a separate VM that the agents can never access. The evaluation pipeline rejects edits outside the permitted files and rebuilds the original and candidate from source. A candidate must pass the repo’s tests and match the original’s output on public and hidden inputs. It must also pass generated tests that only the pipeline holds, which catch submissions that overfit to the existing tests or exploit gaps in them. Timing runs only on the hidden inputs, on a dedicated Intel machine,