Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework

Nils Dycke and Iryna Gurevych found that flaws in research logic have no significant effect on the output of automatic review generators (ARGs), according to a paper published in Transactions of the Association for Computational Linguistics, Volume 14, pages 465–488. The authors present a fully automated counterfactual evaluation framework that isolates and tests the core reviewing skill of detecting faulty research logic — evaluating internal consistency between a paper's results, interpretations, and claims — and release the counterfactual dataset and evaluation framework publicly along with three actionable recommendations for future work.

Abstract Large Language Models LLMs have great potential to accelerate and support scholarly peer review and are increasingly used as fully automatic review generators ARGs . However, potential biases and systematic errors may pose significant risks to scientific integrity; understanding the specific capabilities and limitations of state-of-the-art ARGs is essential. We focus on a core reviewing skill that underpins high-quality peer review: detecting faulty research logic. This involves evaluating the internal consistency between a paper’s results, interpretations, and claims. We present a fully automated counterfactual evaluation framework that isolates and tests this skill under controlled conditions. Testing a range of ARG approaches, we find that, contrary to expectation, flaws in research logic have no significant effect on their output reviews. Based on our findings, we derive three actionable recommendations for future work and release our counterfactual dataset and evaluation framework publicly.1 - Anthology ID: - 2026.tacl-1.22 - Volume: - Transactions of the Association for Computational Linguistics, Volume 14 https://aclanthology.org/volumes/2026.tacl-1/ - Month: - Year: - 2026 - Address: - Cambridge, MA - Venue: - TACL https://aclanthology.org/venues/tacl/ - SIG: - Publisher: - MIT Press - Note: - Pages: - 465–488 - Language: - URL: - https://aclanthology.org/2026.tacl-1.22/ https://aclanthology.org/2026.tacl-1.22/ - DOI: - 10.1162/tacl.a.642 https://doi.org/10.1162/tacl.a.642 - Cite ACL : - Nils Dycke and Iryna Gurevych. 2026. Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework https://aclanthology.org/2026.tacl-1.22/ . Transactions of the Association for Computational Linguistics , 14:465–488. - Cite Informal : - Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework https://aclanthology.org/2026.tacl-1.22/ Dycke & Gurevych, TACL 2026 - PDF: - https://aclanthology.org/2026.tacl-1.22.pdf https://aclanthology.org/2026.tacl-1.22.pdf