Author-in-the-loop Agentic Reproducibility Auditing for a Machine Learning Conference

A team of researchers deployed an author-in-the-loop agentic reproducibility audit across 36 conference papers and 133 experimental tables and figures, raising 20 issues after human review, according to arXiv paper 2610.22465v1. The system uses a structured checklist of reproduction instructions and limitations plus a two-stage GitHub workflow in which an AI agent assesses whether results were reproduced and whether authors need to clarify or repair, while human reviewers approve requests and decide if concerns are resolved. Further checks supported closing all 20 issues, though full reproduction remained unverified in some cases.

arXiv:2610.22465v1 Announce Type: new Abstract: Reproducibility checks during peer review require a shared understanding of what authors' code is expected to reproduce. We present an author-in-the-loop collaborative audit that makes author declarations explicit inputs to an AI agent's investigation and assessment. A structured checklist records reproduction instructions and limitations; a two-stage workflow on GitHub connects agent checks to author clarification, revision, and agent reassessment. The agent separately assesses whether results have been reproduced and whether authors need to provide clarification or make repairs, while human reviewers approve requests and decide whether concerns are resolved. We deployed the system across 36 conference papers and 133 experimental tables and figures, raising 20 issues after human review. Further checks supported closing all the issues, although full reproduction remained unverified in some cases. We further compared repeated audits with complete and blank checklists to examine how author declarations shaped the agent's investigation and assessment. Our work contributes a collaborative workflow and agent design that use author knowledge to guide reproducibility auditing. The conference deployment provides empirical evidence that this approach can help identify and address reproducibility problems through collaboration among authors, agents, and reviewers.