Confident Science Papers Sway AI Judges – Regardless of Merit

A new German research paper found that LLM reviewers consistently score confidently written scientific papers higher than identical papers with less assertive language, in some cases flipping decisions across the accept/reject boundary. The study rewrote 504 papers from 2026 ICLR submissions using Claude Opus 5 to vary confidence while keeping scientific content unchanged, then had six AI models including GPT-5.5, Claude Sonnet 4.6 and Gemini 3.5 Flash assess them, with reviewers more often praising the methods, theories and contributions of the confident versions.

Anderson's Angle https://www.unite.ai/series/andersons-angle/ Confident Science Papers Sway AI Judges – Regardless of Merit Add Unite.AI to your preferred sources on Google https://www.google.com/preferences/source?q=unite.ai Confidence is a distinct advantage in evolution https://www.bath.ac.uk/announcements/why-evolution-made-us-overconfident-the-peacocks-tail-of-human-behaviour/ :~:text=Our%20research%20suggests%20that%20those%20costs%20are%20exactly%20what%20makes%20it%20useful%2E%E2%80%9D – and, according to a new research paper from Germany, also an advantage when a scientific paper is getting reviewed by Large Language Models LLMs https://www.unite.ai/large-language-models/ instead of an author’s peers an increasingly frequent necessity, as the volume of science papers balloons https://www.unite.ai/the-survey-paper-ddos-attack-thats-overwhelming-scientific-research/ thanks to the productivity gains https://www.unite.ai/fighting-ai-slop-in-science-papers/ under AI . The new study finds that AI i.e., non-human reviewers consistently give higher scores to scientific papers that are written in confident, assertive language – even when the underlying research is identical to papers with a less confident tone. In some cases, changing nothing but the wording proves enough to turn a rejection into an acceptance. The authors state: ‘Our experiments reveal substantial preferences of LLM reviewers toward papers using high-confidence language. Crucially, these preferences frequently result in decision flips across the accept< reject boundary, directly affecting review outcomes. ‘Beyond scores, we find that aspects raised in reviews vary in frequency and sentiment depending on linguistic features of the paper, potentially influencing final decisions.’ It’s ironic, because the syndrome that the authors identify is very similar to the way https://www.unite.ai/why-cant-ai-just-admit-that-it-doesnt-know-the-answer/ AI can so often be ‘confidently wrong’ https://www.unite.ai/confidently-wrong-why-the-smartest-ai-models-are-the-worst-at-correcting-themselves/ when responding to our own prompts and queries; except here, the decision is human, and the individual and collective responsibility real, whether or not the contributing scientists behind these papers really believe their own claims. If AI increasingly decides which scientific papers get published, scientists may learn to ‘sound more confident’ as a competitive tactic , perhaps most particularly when their submissions are flawed; and this could allow weaker science an undeserved foothold in the ever-diminishing pool of available attention. Bluster Towards Victory According to the study https://arxiv.org/pdf/2610.35254 , AI reviewers were more likely to praise the methods, theories and scientific contributions of confidently-written papers, even when these were identical to those in less assertive versions. In the authors’ tests, 504 scientific papers were rewritten in different styles, and assessed by six AI models, including GPT-5.5 https://openai.com/index/introducing-gpt-5-5/ , Claude Sonnet 4.6 https://www.anthropic.com/news/claude-sonnet-4-6 and Gemini 3.5 Flash https://www.unite.ai/google-ships-three-gemini-flash-models-as-its-flagship-slips/ . It was necessary to generate artificial/amended versions of papers, so that the eccentricities and quirks of real submissions did not skew the results: The papers were drawn from submissions to the 2026 ICLR conference, comprising equal numbers of accepted and rejected papers. Claude Opus 5 https://www.unite.ai/anthropics-opus-5-nears-frontier-intelligence-at-opus-prices/ then rewrote each submission to sound more or less confident<sup †</sup , keeping the scientific content unchanged, and thus allowing differences in the AI reviews to be attributed to writing style, i.e., a varied level of confidence expressed. In the results, AI reviewers gave higher scores to scientific papers written in confident language, even when the evidence was identical. Claude Sonnet 4.6 showed the strongest bias, favoring confident versions seven times more often than cautious ones. In 21% of cases, it recommended accepting a confidently-written paper, while rejecting a more cautiously-expressed version of the same paper . The new study states: ‘Our findings raise important concerns for the use of LLMs in peer review. If content-equivalent papers receive different scores and review arguments solely because of linguistic properties, reviewer judgments do not exclusively reflect scientific quality. ‘LLM reviewers should therefore be stress-tested on content-preserving rewrites to determine whether linguistic changes systematically affect their assessments.’ The authors go on to suggest that these tests should explicitly examine both the papers’ scores and their written reviews, to establish whether confident language is being mistaken for stronger scientific evidence. Method A total of 504 scientific papers were selected from submissions to the 2026 International Conference on Learning Representations ICLR . The sample was divided equally between accepted and rejected papers, with 24 submissions drawn from each of the conference’s 21 research categories: Each paper was anonymized and rewritten using Claude Opus 5, producing alternative versions designed to test both the confidence with which claims were expressed, and the apparent English-language proficiency of the writer. For the confidence tests, statements were made either more assertive or more cautious. Words suggesting certainty, such as ‘clearly’ and ‘certainly’ , were used more frequently in ‘confident’ versions, while qualifying expressions such as ‘may’ and ‘suggests’ were favored in more circumspect drafts. In every case, the scientific claims, evidence, results, technical terminology and overall structure remained unchanged. The authors note that the rewriting instructions were based on established linguistic research, rather than on general requests to imitate particular writing styles. Checking the Rewrites For checks, each rewrite was compared with its original using semantic similarity https://web.archive.org/web/20261008122807/https://docs.ragas.io/en/stable/concepts/metrics/available metrics/semantic similarity/ , which assesses whether passages retain their meaning; and TF-IDF https://www.geeksforgeeks.org/machine-learning/understanding-tf-idf-term-frequency-inverse-document-frequency/ similarity which measures the preservation of vocabulary and terminology . Both scores approached 1, confirming strong similarity to the originals: Six AI reviewers were tested: proprietary models GPT-5.5; Claude Sonnet 4.6; and Gemini 3.5 Flash; and open-weight models Gemma 3 https://www.unite.ai/gemma-3-googles-answer-to-affordable-powerful-ai-for-the-real-world/ ; Qwen3 VL https://www.unite.ai/alibaba-releases-qwen3-vl-technical-report-detailing-two-hour-video-analysis/ ; and GLM 4.7 https://z.ai/blog/glm-4.7 . These were chosen partly because they all have pre-2026 knowledge cutoff dates https://web.archive.org/web/20260727202546/https:/www.temso.ai/blog/ai-knowledge-cutoff-dates-every-major-llm-updated-for-2026 , reducing the risk of prior knowledge of the papers’ acceptance decisions. Each rewritten paper was reviewed independently using conference-style guidelines. To establish the extent to which scores might vary between repeated assessments, ten randomly-selected papers were reviewed ten times by each model. Results The tests conducted examined whether the six selected AI reviewers would consistently favor particular writing styles, and compared papers written in ‘confident’ or ‘cautious’ language, and papers resembling academic English written by native or non-native speakers<sup †</sup . Confident writing was consistently favored by all six AI reviewers. The strongest preference was shown by Claude Sonnet 4.6, which, as mentioned earlier, awarded higher scores to ‘confident’ drafts seven times more often than to ‘cautious’ ones. A paper was considered accepted if its score reached 4.5 out of 10, though changes in writing style sometimes pushed the same paper above or below this threshold, reversing the decision; and these reversals generally favored confident writing. Claude Sonnet 4.6 performed particularly poorly, accepting the ‘confident’ version while rejecting the ‘cautious’ version in 21% of cases. The reverse occurred in just 2.8% of cases. The effect was also evident in the written reviews, where AI reviewers were more likely to praise a paper’s methods, theories and scientific contributions if its claims were expressed confidently. Since the research itself had not changed, these more favorable assessments could only be attributed to differences in presentation and tone. Conclusion Every time humans use AI to albeit inadvertently increase the signal-to-noise-ratio, AI is trialed as the solution – here, to cope with the great increase in Artificial Intelligence research paper submissions; a surge that has increased the gatekeeping power and parameters of Arxiv https://arxiv.org/abs/2610.33980 lately. Yet it must be clear by now that frontier LLMs, and all the tiers beneath them, still have far too many eccentricities to be generically entrusted as judges, except in the most rigidly parametrized circumstances. In such cases, turning smaller-scale LLMs into dedicated judging tools via fine-tuning https://www.unite.ai/a-full-guide-to-fine-tuning-large-language-models/ or LoRAs https://www.unite.ai/lora-qlora-and-qa-lora-efficient-adaptability-in-large-language-models-through-low-rank-matrix-factorization/ would probably be a better idea. However, that entails at least a little extra effort, if not a little extra investment, whereas ‘vanilla’ local installs or API calls are quicker, and relatively cheap. This rather tersely-written paper may nonetheless be of more than average interest to the LLM end-user, since they will already be familiar with ‘enthusiasm’ as a bugbear that they must maneuver round in order to work effectively with AI; and, if this and similar LLM issues cannot yet be fixed, they can at least be accounted for. Before we take too derisive a tone about this kind of facile equivalence in the estimation of AI, it’s worth remembering that these systems are as lost as the rest of us when it comes to epistemology – the search for definitive authority; we too are, historically, very prone to mistake the signal for the thing itself. Titled ‘Style over Substance? Linguistic Bias in LLM-based Peer Review’ <sup †</sup Or to resemble academic English written by native or non-native speakers – a parallel hypothesis that the paper was not able to prove as effectively, and which is not addressed in this article. First published Tuesday, October 27, 2026