PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews

Researchers introduced ReviewFact, a benchmark that labels factual errors in peer-review comments across NLP, machine learning, and the natural sciences while measuring cost and latency alongside accuracy, and PremiseCheck, a framework pairing an LLM that rewrites each comment into explicit premises without reading the paper with a decision model that checks each premise against the paper. PremiseCheck reaches accuracy comparable to GPT-5.5 across all three fields at about half its cost when each comment is checked on its own, and on NLP reviews at a quarter of the cost and less than half the latency, according to the arXiv:2610.21046v1 paper. The work targets peer review strained by growing submission volume, where human- or AI-written review comments can misstate what a paper says.

arXiv:2610.21046v1 Announce Type: new Abstract: Peer review is under strain from the growth of submissions, and review comments written by humans or AI can misstate what a paper says. We introduce ReviewFact, a benchmark that labels such factual errors in review comments under one definition across NLP, machine learning, and the natural sciences, and that measures cost and latency alongside accuracy. A frontier LLM checks comments well but must read the whole paper for each check, while cheap decision models, which score whether a statement holds in a text, fall far short on whole comments that mix several implicit assertions. PremiseCheck is a framework that combines the two. An LLM rewrites each comment as explicit premises without reading the paper, and a decision model checks each premise against the paper. When each comment is checked on its own, PremiseCheck reaches accuracy comparable to GPT-5.5 across all three fields at about half its cost, and on NLP reviews at a quarter of its cost and less than half its latency.