RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?

A new benchmark called RECLAIM, built from 100 NeurIPS 2025 papers, found that the best AI agent reproduced only 41% of Run-tier papers, 27% at the Retrain tier, and 15% at the Reimplement tier, according to the arXiv paper 2609.28850v1. Each paper fixes in advance the result to reproduce, the success criteria, and a GPU-hour budget, with difficulty set by what authors released (code, data, and weights for Run; no weights for Retrain; no code for Reimplement). Failed attempts used an average of 29% of their budget, and the most common agent error was writing the method without checking any part against the paper's numbers, occurring in 63 of 400 runs.

arXiv:2609.28850v1 Announce Type: new Abstract: Reproducing a machine learning paper involves most research steps, from installing software and debugging to running experiments, work that AI agents increasingly do. We introduce RECLAIM, a benchmark of 100 NeurIPS 2025 papers that can be rebuilt yearly from new conferences. For each paper we fix in advance the result to reproduce, what counts as a successful reproduction, and a GPU-hour budget. An agent must reproduce that result using the paper and whatever its authors released. What the authors released decides the difficulty tier. Run-tier releases include code, data, and weights; Retrain-tier releases lack weights, so the agent trains the model; Reimplement-tier releases lack code, so the agent writes it. A separate language model grades runs from logs and outputs rather than agents' reports. We run four agents once per paper; the best agent in each tier reproduces only 41% of Run-tier papers, 27% at Retrain, and 15% at Reimplement, where every agent does worst. Failed attempts use on average 29% of their budget, so most stop with budget left. The most common agent error is writing the method without checking any part against the paper's numbers, in 63 of 400 runs.