Two arXiv v1 preprints, SwarmEvolve and Scientist World, measure different research-agent tasks: evolutionary program search and performance in paper-derived environments.
Throughput and Evolutionary Search
SwarmEvolve, authored by Qile Jiang and George Em Karniadakis, focuses on the efficiency of evolutionary program search. The system operates by deploying 10 swarms, each containing 45 latent agents per round. To maintain diversity, a novelty filter keeps 30 agents in each swarm; summarizers read their KV caches to synthesize candidate programs. This mechanism is tested across 21 mathematical and scientific-computing problems.
The researchers tested Qwen3 8B, 14B and 32B-FP8 models, using three seeds and 60-minute runs on one GPU with eight CPU cores. Against OpenEvolve, ShinkaEvolve and CodeEvolve, the authors report the highest average score across all three model sizes, 5 to 13 times more programs evaluated per hour, and median times of 3 to 8 minutes to reach the baselines’ 60-minute scores. The reported measures combine program quality with search throughput: one describes results, the other the number of candidate programs the system can evaluate under the fixed time budget. These are author-reported benchmark outcomes, not an independent replication. The denominator here is the set of 21 specific problems, and the performance is bounded by the 60-minute compute window and the specific hardware configuration of one GPU and eight CPU cores.
Performance in Paper-Derived Environments
Scientist World, by Taiming Lu, Sachin Konan and Zhuang Liu, shifts attention from search throughput to agent performance within paper-derived environments. The authors released 1,000 executable environments derived from research papers, but the model evaluation uses a fixed cohort of ten lower-compute environments, not the full release. That denominator matters: 84% and 74% describe attempts within this selected cohort and protocol, not a success rate across all scientific work or all 1,000 environments.
The study separates two modes. In improve mode, an agent starts from the reproduced method from a paper and tries to improve it. In innovate mode, it receives the problem without the source code and must devise a method. The reported score measures improvement over the reproduced result, with paper-specific measures aggregated for each environment. The authors report agents beating the reproduced method in 84% of improve attempts and 74% of innovate attempts. The distinction is useful: these modes ask different questions, even within the same environment, and the paper treats them as separate abilities rather than one interchangeable score.
Scientist World also evaluates on visible and held-out slices, so the method developed against visible results is also checked on data kept back for final scoring. That design helps test whether a reported gain carries over beyond the visible slice. It does not remove the cohort boundary: the reported ten environments remain a specific lower-compute sample, and the results remain those of the paper’s own evaluation.
What the Numbers Can Tell Builders
The two papers do not offer competing measurements of one common capability. SwarmEvolve asks how an evolutionary search framework performs on a defined set of program-optimization tasks under fixed model and compute conditions. Scientist World asks whether agents improve reproduced methods or innovate from problem statements in a set of executable research environments. The first includes a throughput measure; the second reports the share of attempts that beat the reproduced methods. A percentage of successful attempts cannot be ranked against programs evaluated per hour as if one were a higher or lower value on the same scale.
For builders allocating model calls, compute or evaluation time, the practical question is what the intended system must do. If the work is program search with an evaluator, throughput and final program scores are relevant. If it is solving paper-derived tasks, an evaluator must distinguish improving a starting method from starting without one, and should show whether gains persist on held-out cases. Neither benchmark, by itself, answers both questions. The task definition determines what the score can support and where it stops.
Evidence Boundary
Both studies are arXiv v1 preprints and report their authors’ own results. The available evidence does not establish independent replication or deployment performance in working scientific labs. Their benchmarks offer bounded evidence about the tasks and experimental settings they describe, not a general measure of scientific autonomy. The useful comparison is therefore not which percentage or throughput figure is larger, but which research-workflow question each setup can actually answer. Builders should view these metrics as specific tools for specific problems, rather than universal indicators of progress.