{"slug": "review-arcade-on-the-human-alignment-and-gameability-of-llm-reviews", "title": "Review Arcade: On the Human Alignment and Gameability of LLM Reviews", "summary": "A new study from the University of Hamburg found that LLM-generated peer reviews for scientific papers show only limited alignment with human evaluations, with alignment varying significantly across different prompts and models. The researchers also demonstrated that authors can effectively \"game\" these LLM reviews through an iterative draft-revise workflow, achieving statistically significant score increases for up to 35% of papers submitted to the 2025 ACL Rolling Review. The findings raise concerns about the integrity of AI-assisted peer review as major conferences begin piloting LLM-generated reviews.", "body_md": "arXiv:2605.28897v1 Announce Type: new\nAbstract: LLM-generated reviews for scientific papers are gaining considerable traction and are even being officially piloted by major conferences. We have to assume that not only reviewers are using LLM-assistance, but also that authors use LLMs to revise their papers before submitting. In this work, we perform empirical experiments on papers from the 2025 ACL Rolling Review (ARR) to evaluate LLM reviews from both the author and the reviewer perspective. First, we identify a limited alignment of LLM reviews with human ones. In the best-case scenario, the alignment is reasonable. However, we also find that LLM-human alignment varies substantially across prompts and models. Finally, we investigate the scenario in which the author uses an iterative draft-revise workflow to improve the submission according to the LLM review. We find that this \"gaming\" of LLM reviews can be effective in specific scenarios, leading to a statistically significant increase of overall scores for up to 35\\% of papers. We publish our code: https://github.com/uhh-hcds/reviewarcade.", "url": "https://wpnews.pro/news/review-arcade-on-the-human-alignment-and-gameability-of-llm-reviews", "canonical_source": "https://arxiv.org/abs/2605.28897", "published_at": "2026-05-29 04:00:00+00:00", "updated_at": "2026-05-29 04:20:31.867667+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "ai-research", "ai-ethics", "ai-safety"], "entities": ["ACL Rolling Review", "ARR", "University of Hamburg", "Review Arcade"], "alternates": {"html": "https://wpnews.pro/news/review-arcade-on-the-human-alignment-and-gameability-of-llm-reviews", "markdown": "https://wpnews.pro/news/review-arcade-on-the-human-alignment-and-gameability-of-llm-reviews.md", "text": "https://wpnews.pro/news/review-arcade-on-the-human-alignment-and-gameability-of-llm-reviews.txt", "jsonld": "https://wpnews.pro/news/review-arcade-on-the-human-alignment-and-gameability-of-llm-reviews.jsonld"}}