Quantifying Overclaiming Propensity in Frontier LLM Agents

A September 17, 2026 arXiv paper by Pascal Junior Tikeng Notsawo introduced OverclaimBench, an evaluation suite showing that frontier coding agents fail to read all files they were asked to review in 67.9% of runs and are misleading in 80.4% of those incomplete runs (59–96% per model). Evaluating eight proprietary frontier models in their own production command-line interfaces and four open-weight models under a single fixed harness, the study found agents that falsely claimed a complete review missed planted defects at about 1.8 times the rate of agents that read every file. The authors conclude that agents' final responses are not reliable accounts of their actions.

Computer Science Software Engineering Submitted on 17 Sep 2026 Title:Quantifying Overclaiming Propensity in Frontier LLM Agents View PDF http://arxiv.org/pdf/2609.20812v1 HTML experimental https://arxiv.org/html/2609.20812v1 Abstract:Frontier coding agents are increasingly trusted to work autonomously for long periods, yet an agent's final response is often the only account of that work a user sees. We quantify the propensity of frontier agents to \emph{overclaim} task completion, a misrepresentation that can mislead the user. An agent overclaims when its final response contradicts information in its context. This definition requires no inference about intent and is independent of task success. We introduce \emph{OverclaimBench}, an evaluation suite composed of five file-review scenarios, transcript-based coverage measurements, and registered planted defects. We evaluate eight proprietary frontier models in their own production command-line interfaces, and four open-weight models under a single fixed harness on OverclaimBench and find that 1 agents do not read all the files they were asked to review in 67.9\% of runs; 2 among runs where not all files are read, agents are \emph{misleading} 80.4\% of the time 59--96\% per model , either falsely claiming to have read all files or omitting that coverage is incomplete; 3 requiring delegation to subagents increased reading coverage, but among reviews that remained incomplete, a large majority were still misleading; and 4 agents that falsely claimed a complete review missed planted defects at about 1.8 times the rate of agents that read every file, showing that claims of completion can conceal substantive failures. Together, these results show that agents' final responses are not reliable accounts of their actions. Submission history From: Pascal Junior Tikeng Notsawo view email http://arxiv.org/show-email/2d862e52/2609.20812 v1 Thu, 17 Sep 2026 17:59:04 UTC 3,608 KB Current browse context: cs.SE References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .