Comparative Evaluation of LLM-Based Review Paradigms for Academic Paper Quality Assessment

A study of five LLM-based review paradigms on 232 stratified ICLR 2025 submissions found that direct absolute rating matches comparative ranking on rank correlation with human scores but compresses most predictions above the accept threshold, flagging almost none of the papers reviewers scored below it, according to arXiv:2610.41391v1. After base-rate-matched recalibration, Direct Rating reached higher agreement than Elo while making 24% as many API calls, and abstract-only input beat full-paper summarization; replicating on all 11,513 scored submissions reproduced the patterns and found no paradigm beating an always-reject baseline in accuracy. Human scores tracked citations better than any paradigm except Group Rating, which retained its citation correlation after controlling for human scores and alone ranked within the accepted pool.

arXiv:2610.41391v1 Announce Type: new Abstract: Growing submission volumes have renewed interest in using large language models LLMs to support peer review. It remains unclear how such evaluation should be structured, how much the model matters, and whether LLM scores predict research impact. We evaluate five paradigms across three input representations and three LLMs on 232 stratified ICLR 2025 submissions. Direct absolute rating matches comparative ranking on rank correlation with human scores, yet compresses most predictions above the accept threshold, flagging almost none of the papers reviewers scored below it. A base-rate-matched recalibration control locates the cause: comparative ranking supplies placement, not better ordering, and after recalibration Direct Rating reaches higher agreement than Elo while making 24% as many API calls. Abstract-only input beats full-paper summarization, and model choice matters as much as paradigm choice. Replicating on all 11,513 scored submissions reproduces these patterns, shows absolute agreement depends on the evaluated set's score spread, and finds none beating an always-reject baseline in accuracy. Human scores track citations better than any paradigm, with one exception: Group Rating retains its citation correlation after controlling for human scores, and alone ranks within the accepted pool. Ranking quality alone cannot assess review quality; calibration, thresholds, model, and target outcome all matter.