AI-Driven Scientific Progress Threatened by a Strange Obsession A new US study of 2,000 papers from four conferences found that newer AI peer-review models draw roughly 50-70% of their criticisms from appendices, compared with less than 30% for the older GPT-5, and that the bias carries into human reviews. The study attributes the shift to OpenAI's GPT-6.1-Sol and GPT-6-Astra, both released in 2026, which rely heavily on appendix material at every conference; LLMs attribute more than 50% of weaknesses to appendix-related material versus less than 10% for humans, and all 200 LLM-generated reviews for ICLR 2027 included at least one appendix criticism. The researchers report that AI reviewers were more than seven times as likely as humans to flag appendix weaknesses and more than three times as likely to raise criticisms concerning only supplementary material, a pattern that held across API, Codex and web access methods even though Codex does not read the middle of an average-length PDF at all. Anderson's Angle https://www.unite.ai/series/andersons-angle/ AI-Driven Scientific Progress Threatened by a Strange Obsession Add Unite.AI to your preferred sources on Google https://www.google.com/preferences/source?q=unite.ai Key PointsNewer AI reviewers fixate on appendices while human reviewers focus on a paper's main… View more - Newer AI reviewers fixate on appendices while human reviewers focus on a paper's main findings. - Language models treat every passage as equally important so they miss which sections actually matter. - Appendix bias persists across access methods and yields surface-level feedback that barely helps authors improve. A new US study suggests that AI models reviewing scientific papers are developing a strange obsession with the papers’ appendices rather than the main body of text, often paying more attention to minor supporting details than to the main findings. With language models increasingly being used https://www.unite.ai/the-rise-of-ai-in-scientific-discoveries-can-ai-truly-think-outside-the-box/ to assist or even replace human reviewers, the newly-identified tendency appears to be spreading into human peer review, potentially hindering progress in AI research: In a test of 2000 science papers drawn from four popular conferences, the researchers of the new work found that newer models draw around 50-70% of their criticisms from appendices, compared with less than 30% for the older GPT-5. Two of the most prominent newer AI models that are apparently driving even human reviewers towards the appendices and away from the more important main results are OpenAI’s GPT-6.1-Sol, and GPT-6-Astra, both released in 2026. The paper states : ‘The difference between the generations is large. For GPT-5, only a small share of weaknesses relate to the appendix. Both newer models, GPT-6.1-Sol and GPT-6-Astra, rely on it much more heavily, and this holds for every conference. ‘This suggests that the bias towards the appendix is mainly a property of newer models . Even though the new models are an improvement, their bias toward the appendix has grown. Being able to read a paper in detail has not taught them which parts matter most.’ The study also found that this fixation on the appendix differs across the methods by which the AI receives the paper, since API, codex and web interfaces do not assess PDFs the same way – and Codex does not read the middle of the average-length PDF at all, sampling it later only in ‘bites’, if necessary. However, all methods were found to favor the appendix material. Until recently, human and AI reviews did not share this obsession: ‘ Human reviewers treat the main body and the supplementary material very differently from LLMs. LLMs attribute more than 50% of their weaknesses to appendix-related material, while humans attribute less than 10%. ‘The difference also shows at the individual-review level. Of the 200 papers sampled each year from 2021 to 2024, only 159 in total received a human review with any appendix-related weakness. In contrast, all 200 LLM-generated reviews for ICLR 2027 included at least one appendix criticism.’ As shown in the test table above, AI reviewers were more than seven times as likely as humans to identify weaknesses in the appendices, and more than three times as likely to raise criticisms concerning only the supplementary material – an indication that some of the latest frontier AI models are now prioritizing details that human reviewers would traditionally regard as secondary to a paper’s main scientific contribution. Why? As far as can be discerned this is the first study to have identified the syndrome , part of the problem is that the AIs do not recognize, or have not gleaned from their copious training material, that the appendix of a paper is not where the payload is normally placed; their general tendency to treat all text as equally important omits the context for appendix material, which is often uncurated and much more roughly-presented than the main body: ‘LLMs almost never criticize the opening pages, where a paper frames its problem and argues why it matters. Humans often push back on exactly this framing, while LLMs seem to accept it and move on to the numbers. ‘ Humans and LLMs read the appendix very differently. Human criticisms fade quickly as the appendix goes on, and almost disappear in its last pages. LLM weaknesses do not fade at all. In fact, LLM weaknesses peak well into the appendix rather than at its start, and they continue even beyond page 30, where human reviewers raise almost no concerns. ‘For an LLM, a detail on page 25 is as fair a target as the main results table.’ The authors note that LLMs’ egalitarian approach to text may naturally have led them to concentrate more on appendix material, because, as stated above, that material tends to have more errors. Therefore the high-level view that I take from this new study is that LLMs have honed in over time on a section of new science papers where a large degree of error or conflation tends to occur, simply because that area is treated so often by human authors as an untidy dumping ground of secondary proof and evidence. For this reason, AI increasingly finds the appendices more interesting than the highly-curated main body, because there is more for it to analyze and correct there ; consequently, the paper notes, LLMs tend to treat the front section of the paper as factual reporting, taking it at face value: ‘Humans expect the main body to stand on its own, so they object when essential sections are pushed into the appendix. LLMs almost never raise this concern. ‘This highlights a key difference: humans recognize that the main body carries more weight than the appendix, while LLMs do not. ‘ Consequences and Remedies This unexpected focus on minor details could have serious implications for AI-assisted scientific research, particularly if the importance of a paper’s main findings is overlooked. Since the tendency appears to have emerged only with the latest models, its longer-term effects are not yet known. The paper acknowledges that valuable errors can be caught by AI reviewers, even when those errors are buried deep in an appendix. However, it warns that too much attention can be paid to minor details at the expense of a paper’s main findings, leaving to lengthy reviews of little practical value: ‘LLM reviewers raise the standard, since they leave no part of the paper unexamined. However, effective reviewing also requires judgment about what matters most. When most of a review’s weaknesses come from appendix content unrelated to the main contribution, the feedback stays on the surface. It does not engage with the core of the paper and does little to help authors improve it. ‘ The authors of the new work https://arxiv.org/pdf/2611.17811 , titled AI Reviews Are Biased Towards the Appendix , were inspired to pursue the thread because they noticed certain odd patterns in the way that LLM-assisted/generated reviews would evaluate their own submissions. The study’s code is available at GitHub https://github.com/cincynlp/review appendix . Method and Tests For the tests, four leading AI models from OpenAI and Anthropic were selected: GPT-6-Astra https://www.unite.ai/openai-releases-gpt-6-astra-its-first-model-rated-critical-for-cyber/ ; GPT-6.1-Sol https://www.unite.ai/openai-unveils-gpt-6-1-sol-at-devday-with-new-codex-and-chatgpt-tools/ ; Claude-Opus-5.5 Claude-Opus-5.5 ; and Claude-Sonnet-5.5 Claude-Sonnet-5.5 , with the models accessed through Codex https://www.unite.ai/openai-codex-review/ and Claude Code https://www.unite.ai/anthropic-brings-claude-code-power-to-everyone-with-cowork/ , and their reviewing behavior compared against API and web-interface access. Research papers were collected from four major AI and natural language processing conferences: ACL 2026 https://2026.aclweb.org/ ; EMNLP 2026 https://2026.emnlp.org/ ; COLM 2026 COLM%202026 ; and ICLR 2027 https://iclr.cc/ – all chosen because they share similar submission and review formats, with strict page limits on the main paper, but generally unrestricted appendices placed after the references. A total of 2,000 papers were examined, comprising 1,200 from ICLR 2021–2026, and 800 from four recent conferences, with 200 selected from each . The 800 recent papers all featured appendices of at least 15 pages. The ICLR 2027 submissions were already anonymous, while authors’ names, affiliations, email addresses, and identifying links were removed from the ACL, EMNLP, and COLM papers to prevent their authorship from being revealed to the AI reviewers. Internet access was disabled for the models, ensuring that the reviews would be based entirely on the PDFs provided. First, the AI was asked to produce a review in the style of the relevant conference, covering the paper’s strengths, weaknesses, and an overall score, with the instructions kept deliberately simple, to reflect how AI might be used by an actual reviewer. The paper and its completed review were then given back to the same AI model, which was asked to identify which section of the paper each criticism referred to. This was determined by examining the criticism alongside the original paper: By separating these tasks, the parts of the papers that attracted criticism could be examined without the original reviews being influenced by additional instructions. Among the results that we have not discussed so far is the authors’ test of where the weaknesses in LLM-generated reviews originate: As shown above, about half or more of the criticisms made by all four AI models were based on material in the appendices – a pattern found across all four conferences. Some criticisms were directed exclusively at the appendix, with no connection to the main paper, accounting for 17.33% of the weaknesses identified by GPT models, compared with 10.16% for Claude. The results, the paper suggests, indicate that appendix material was being given disproportionate attention, with relatively minor details sometimes treated as grounds for criticism of the paper as a whole . A comparison across methods was conducted to establish whether the appendix bias was affected by how the AI was accessed. Using GPT-6-Astra, one hundred ICLR 2027 submissions were reviewed via Codex; the ChatGPT web interface; and an API supplied with either the original PDF or its extracted text. With Codex, text was extracted and examined through commands; through the API, either the complete PDF or its extracted text was supplied directly; while uploaded documents were processed through the web interface’s own opaque system: As shown above, appendix-related material was cited in 62.2% to 73.7% of criticisms across the four methods, with the highest proportion recorded when extracted text was supplied directly through the API. The bias was therefore found to persist regardless of how the model was accessed. The bulk of the other tests were already reported earlier in the article, and we refer the reader to the source material for further information. Submission Shenanigans Much as the subject is personal to the authors of the new work, so it is to me, since I read scientific papers a lot, and have learned the hard way the hidden rules, and subversion of rules, that dictate the way papers are composed and presented. AI is not broadly at fault for paying attention to the appendices, even if it is doing so for the wrong reasons, since authors often bury the paper’s ‘gotchas’ https://www.unite.ai/towards-automated-science-writing/ :~:text=Other%20%E2%80%98Gotchas there – factors or results that may all but nullify, or else severely qualify the claims made in the main body of the paper. Neither are authors necessarily at fault for shunting important material to the appendix, since the templates of popular outlets such as those featured as data sources in the new study prescribe maximum page or word-count limits https://icml.cc/Conferences/2026/AuthorInstructions :~:text=length%20limit%20of%20the%20paper%20body%20is%209%20pages on the main body of the paper. When reporting complex study results, many authors clearly find this a severe restriction; in what I would consider ‘the worst cases’