AI reviews are the new normal

A MathArena study by Dekoninck, Gehrunger, Rögnvaldsson, Sun and Vechev found that GPT-6.1 Sol reported major errors in about 6% of 656 arXiv papers from January 2025 to July 2026, but in only 1.5% of 137 papers from August and September 2026, a period when over a quarter of arXiv papers began acknowledging AI use. The authors suggest the drop likely reflects authors running AI checks and fixing errors before posting, and recommend an AI check before posting a manuscript and on older papers. The findings were published in the October 2026 MathArena write-up "Searching for errors in mathematical papers with GPT-6.1 Sol.

SERIES · AI is making papers cheap to produce but they remain costly to referee. Proofs and Prompts is publishing a series of posts from 26 October on the future of publishing, papers and credit. Questions such as how to disclose AI use, who bears the cost of checking, what authorship means and what papers of the future might look like will be addressed. This post is part of the series. Each is written independently and reflects only its authors’ views. All posts in the series The mathematical literature contains mistakes. Articles on the arXiv are not refereed, and even peer review does not guarantee that articles are error-free. In private communication, many tenured and non-tenured colleagues have told me that they now run their drafts through an AI model before posting, and that this sometimes catches serious mistakes. The capabilities of AI to catch mistakes seem to have improved drastically in the last months. In a recent study for MathArena, we asked GPT-6.1 Sol to extract the main theorems of arXiv papers and search for explicit counterexamples to these theorems. Each counterexample was then verified and classified by severity. Across 656 papers from January 2025 to July 2026, the model reported major errors in about 6% of papers. For 137 papers from August and September 2026, a period in which over a quarter of arXiv papers began acknowledging AI use, that rate fell to 1.5%. A likely explanation is that some authors already ran an AI check and fixed what it found. Part of the effect may also be because more recent work is partly AI-generated, and a model may have a harder time finding errors in an AI-generated argument. Still, the trend matches what people are telling me and my personal experience. Our suggestion is to do an AI check before posting a manuscript, and there is evidence that many mathematicians are already doing so. Also, it is very likely good practice to run your older papers through an AI check to see if any results need fixing. Chart from the MathArena study by Dekoninck, Gehrunger, Rögnvaldsson, Sun and Vechev MathArena, October 2026 . Full write-up and data: J. Dekoninck, T. Gehrunger, K. Rögnvaldsson, C. Sun, M. Vechev, Searching for errors in mathematical papers with GPT-6.1 Sol, MathArena, October 2026. https://matharena.ai/literature mistakes/ “Our suggestion is to do an AI check before posting a manuscript” An alternative would be to stop intentionally looking away from the disastrous societal and environmental impacts these fascist-adjacent companies are inflicting, or from their abusive relationship with the mathematical community, and refrain from using these models. That also seems like “very likely good practice”. I completely agree with JPQ on this. Why so many members of our community choose to engage with the products of people who evidently don’t care at all for us is completely beyond me not to mention the fact, as JPQ points out, that they also don’t care at all for society as a whole, nor for the environment . Could you concretely explain why you believe companies like OpenAI and Anthropic are fascist-adjacent? Dario Amodei’s essays in his blog literally firstname lastname dot com covers a lot of his thought, none of which seems fascist to me and explicitly addresses societal concerns, political concerns, economic concerns e.g. equitable development across marginalized nations , environmental concerns, and various other concerns. Anthropic in particular, and tangential groups i.e. groups that have been broadly written down as effective altruists are also unusually concerned with global welfare, animal rights classically shimp welfare , sustainable development, etc which are very clearly not right-wing values nor adjacent. Why fascist? One of the core parts of fascism is the intense cooperation between industry and state, which is what we see now in the US. We also see AI “Gleichschaltung” in all organizations and AI advertising has reached Goebbels levels at least in its ubiquity and intensity. Can you even go on the Internet for 15 min without reading “AI is awesome”? I think the somewhat lighthearted term “AI fascism” just focuses on these aspects and does not necessarily include racism etc., though some AI CEOs explicitly supported the ICE shootings in Minneapolis. The research methodology is extremely problematic to say the least. It is an almost fully automated LLM pipeline where each stage potentially introduces serious errors. And if we look at the errors that were found these mostly seem to be in CS/Engineering fields e.g. robot-motion, traffic scheduling and LLM training algorithms. One really wishes the authors had followed their own advice and had an AI review the report/paper first. I agree that it would have been ideal to do this by hand, but going through so many articles would have been unfeasible for us to do. While we do not claim that the LLMs found all errors, we manually looked at some of the errors it identified and were generally quite satisfied with how the system worked. In terms of articles for which the system reported errors, roughly 10% of them are from articles with primary categories math.AP and math.OC each, with the next frequent categories after that being math.NT, math.NA, math.DS, math.PR, math.CO, math.DG and finally cs.LG. When one only looks at major errors, the three categories with the most of them are math.OC, math.NT and math.ST although with only 41 articles with reported major errors, the sample size is much smaller of course . Sure doing this entirely by hand would not be feasible, but given the rather small sample sizes of minor and major errors, one should at least run the pipeline multiple times. With stochastic models, one would expect significant deviations between runs—as is already hinted at by the confusion matrix with Claude and the 10% downgrade rate when you reran the 75 major errors. Ideally, this variance should be measured across each individual step in the pipeline theorem extraction, counterexample search, and verification , as all of them are potential sources of compounding and systematic errors. One also needs to account for the rather different standards for what counts as a theorem/proof across fields e.g: cs.LG or math.OC vs. math.NT -some mathematicians would be shocked if they saw some of the things that pass as theorems/proofs in CS/ML. The same goes for how amenable different fields are to finding counterexamples. In short one would need to at least check whether the proportion of arXiv categories differ between the two samples before reaching broad conclusions of the type you present. While the author of this post is talking about users asking AI for feedback/to check their papers, as a follow up, I would like share this paper https://arxiv.org/abs/2605.03202 which explains why journals should not automatize peer review… Very problematic in many aspects, to say the least. You should at least have asked all authors of any single paper of the stack you have scanned that way, to get their consentment for such an intrusive inquiry. Did you ever understand the contents and the results of each paper, and did you understand the answers produced by the LLM in each case? Are we witnessing the birth of an AI-assisted police of mathematical research? To sum up, I am really shocked, and I guess I am not alone. There’s also this paper: Gary King, Steven Worthington. 2026. “AI Article Reviews Dominate Human Peer Reviews at Their Best: Evidence and Policy Choices”. We show that AI-generated article reviews are now better, by every measure we could construct, than reviews written by well-compensated human experts and judged to be of unusually high quality by journal editors and authors. We then discuss how this result may inform journal policy choices, with concerning downsides as well as potential paths to increase publication speed and quality; reduce adversarial interactions; improve the experience of editors, reviewers, authors, and readers; and advance science. My worry is that they do not clarify what does “rewriting the human reviews in the AI’s style” mean. This is about matching the statistics of the machine learning model, and this matters a lot due to the results of “Stop Automating Peer Review Without Rigorous Evaluation” about paper laundering.