Metag: A dataset to build agentic meta-reviewing capabilities

Microsoft researchers introduced Metag, a dataset of 349 action items linking reviewer concerns, author resolutions, and manuscript diffs to help build AI agents that verify whether authors addressed peer-review feedback. The dataset, collected from OpenReview discussions and manuscript versions before and after acceptance, aims to reduce the burden on meta-reviewers by enabling automated identification of changes in scientific articles. It is publicly available at https://github.com/microsoft/Metag-dataset.

arXiv:2608.20488v1 Announce Type: new Abstract: AI tools increasingly support tasks across the scientific research cycle, from experiment design and manuscript preparation to peer review. At the same time, the continuing growth in conference submissions has increased the burden on meta-reviewers, who must synthesize reviewer feedback, author rebuttals, and manuscript revisions. To address this concern, this paper introduces Metag, a dataset to accelerate the development of meta-reviewing agents, specifically to identify changes made to scientific articles during the review-rebuttal process. Each instance contains a reviewer concern, the author's proposed resolution, and the manuscript diffs implementing the stated change. Metag is collected by obtaining manuscript versions from before the review deadline and after acceptance, computing differences between the two documents, and asking human annotators to align these differences with action items from OpenReview discussions. The resulting dataset consists of 349 high-quality action items tied to paper differences and will enable building methods to empower meta reviewers to quickly identify whether authors have addressed reviewer statements and where in the paper those changes have been made, resulting in additional transparency and traceability throughout peer review. The dataset is publicly available at https://github.com/microsoft/Metag-dataset.