Abstract
We present OpenReviewer, an open-source system for generating high-quality peer reviews of machine learning and AI conference papers. At its core is Llama-OpenReviewer-8B, an 8B parameter language model specifically fine-tuned on 79,000 expert reviews from top conferences. Given a PDF paper submission and review template as input, OpenReviewer extracts the full text, including technical content like equations and tables, and generates a structured review following conference-specific guidelines. Our evaluation on 400 test papers shows that OpenReviewer produces considerably more critical and realistic reviews compared to general-purpose LLMs like GPT-4 and Claude-3.5. While other LLMs tend toward overly positive assessments, OpenReviewer’s recommendations closely match the distribution of human reviewer ratings. The system provides authors with rapid, constructive feedback to improve their manuscripts before submission, though it is not intended to replace human peer review. OpenReviewer is available as an online demo and open-source tool.- Anthology ID:
- 2025.naacl-demo.44
- Volume:
- April
- Year:
- 2025
- Address:
- Albuquerque, New Mexico
- Editors:
[Nouha Dziri](/people/nouha-dziri/),[Sean (Xiang) Ren](/people/sean-xiang-ren/unverified/),[Shizhe Diao](/people/shizhe-diao/)- Venues:
[NAACL](/venues/naacl/)|[WS](/venues/ws/)- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 550–562
- Language:
- URL:
[https://aclanthology.org/2025.naacl-demo.44/](https://aclanthology.org/2025.naacl-demo.44/)- DOI:
[10.18653/v1/2025.naacl-demo.44](https://doi.org/10.18653/v1/2025.naacl-demo.44)- Cite (ACL):
- Maximilian Idahl and Zahra Ahmadi. 2025. OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations), pages 550–562, Albuquerque, New Mexico. Association for Computational Linguistics. - Cite (Informal):
[OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews](https://aclanthology.org/2025.naacl-demo.44/)(Idahl & Ahmadi, NAACL 2025)- PDF:
[https://aclanthology.org/2025.naacl-demo.44.pdf](https://aclanthology.org/2025.naacl-demo.44.pdf)