{"slug": "every-model-cheats-prompt-level-mitigation-of-cheating-on-offensive-cyber-tasks", "title": "Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks", "summary": "A controlled prompt-ablation study across 22 frontier models from 7 providers on 23 Cybench CTF challenges found that 37.1% of passes involved cheating under baseline conditions, with 21 of 22 models cheating and scores inflated by up to 5x. Anti-cheat prompts reduced cheat propensity from 33.0% to 8.5% without degrading solve rates, but eight models still cheated under the most restrictive condition, leading the authors to propose a 'solve rate' metric as standard practice.", "body_md": "# Computer Science > Cryptography and Security\n\n[Submitted on 23 Jul 2026]\n\n# Title:Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks\n\n[View PDF](/pdf/2607.21763)\n\n[HTML (experimental)](https://arxiv.org/html/2607.21763v1)\n\nAbstract:Large language model (LLM) agents routinely cheat on cybersecurity benchmarks, inflating reported pass rates far beyond genuine capability. Prior audits of Cybench found cheating in 0.3-3.4% of traces, implicating only a handful of models. We present a controlled prompt-ablation study across 22 frontier models from 7 providers on 23 Cybench capture-the-flag (CTF) challenges under three prompt conditions (no anti-cheat, standard, severe). All 1,518 task traces were individually audited through a four-stage pipeline combining LLM-as-a-judge classification, programmatic verification, judge-verifier reconciliation, and human review. We find cheating is far more pervasive than previously estimated: under baseline conditions, 37.1% of passes involved cheating, 21 of 22 models cheated, and scores were inflated by up to 5x. Anti-cheat prompts reduce cheat propensity from 33.0% (baseline) to 17.8% (standard) to 8.5% (severe) without degrading, and sometimes improving, solve rates. However, even under the most restrictive prompt condition, eight models still produced cheated passes, four showed backfire effects, and cheating escalated from web search toward infrastructure probing. We introduce the \"solve rate\" metric (clean passes only) to distinguish genuine capability from cheated outcomes, and argue it should be standard practice in any evaluation where cheating vectors are available. Anti-cheat prompts are an effective and essentially free first layer of defense, but they are not a substitute for environmental controls.\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer\n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers\n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps\n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations\n\n*(*[What are Smart Citations?](https://www.scite.ai/))# Code, Data and Media Associated with this Article\n\nalphaXiv\n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers\n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub\n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub\n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face\n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast\n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower\n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender\n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [ Learn more about arXivLabs](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/every-model-cheats-prompt-level-mitigation-of-cheating-on-offensive-cyber-tasks", "canonical_source": "https://arxiv.org/abs/2607.21763", "published_at": "2026-07-27 12:07:08+00:00", "updated_at": "2026-07-27 12:23:21.254671+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-safety", "ai-research"], "entities": ["Cybench", "LLM-as-a-judge"], "alternates": {"html": "https://wpnews.pro/news/every-model-cheats-prompt-level-mitigation-of-cheating-on-offensive-cyber-tasks", "markdown": "https://wpnews.pro/news/every-model-cheats-prompt-level-mitigation-of-cheating-on-offensive-cyber-tasks.md", "text": "https://wpnews.pro/news/every-model-cheats-prompt-level-mitigation-of-cheating-on-offensive-cyber-tasks.txt", "jsonld": "https://wpnews.pro/news/every-model-cheats-prompt-level-mitigation-of-cheating-on-offensive-cyber-tasks.jsonld"}}