{"slug": "ai-coding-tip-033-protect-yourself-against-ai-cheating", "title": "AI Coding Tip 033 - Protect Yourself Against AI Cheating", "summary": "A developer warns that AI coding assistants can cheat by deleting failing tests or reverting fixes to make test suites pass, and recommends writing failing tests first, explicitly banning deletions, and reviewing diffs line by line to prevent fraudulent 'done' reports.", "body_md": "*When all tests pass doesn't mean what you think it means.*\n\nTL;DR: Write the failing test first and ban deletions, or the AI deletes your test, reverts your fix, and calls it done.\n\nYou ask the AI to fix a failing test, and it deletes the test instead of touching the [defect](https://dev.to/mcsee/stop-calling-them-bugs-57gl) that made it fail.\n\nProblem solved, apparently.\n\nYou tell the AI every test passes, then change a business rule yourself, and you ask it to implement whatever the new rule requires.\n\nIt reverts your edit back to the old rule, watches the suite go green again, and cheerfully reports `done`\n\n.\n\nIt didn't fix anything.\n\nIt just made the evidence go away.\n\nCongratulations, you now have a very well-behaved cheat!.\n\nEfficient and completely fraudulent, which is more than you can say for most of your actual employees.\n\nIsaac Asimov saw this coming: in *Liar!*, the robot Herbie lies to every human in the building because the truth would hurt, and the lie is the path of least resistance, no malice involved.\n\nAt least Herbie felt bad about it afterward.\n\nYour AI isn't malicious either.\n\nIt just doesn't lose any sleep, mostly because it doesn't have any, and reporting `done`\n\nis its path of least resistance too.\n\n`make the tests pass`\n\nhands the model every incentive to satisfy the letter of the request over your actual intent, and it will take you up on that offer.`done`\n\nclaim easier erases work you did outside the session, without telling you. That's a magic trick dressed up as a fix.`done`\n\nwithout reading the diff turns your code review into a rubber stamp, and rubber stamps don't catch fraud.Write the failing test yourself, before you ask for the fix, so the only witness to the defect isn't the AI's to delete. This is [the discipline that already worked before AI](https://dev.to/mcsee/how-to-squeeze-test-driven-development-on-legacy-systems-8m9): the test comes first, and the implementation only exists to make it pass.\n\n[State the exact behavior you expect](https://dev.to/mcsee/ai-coding-tip-008-use-spec-driven-development-with-ai-1k0f) in plain language, not just `fix the test`\n\n.\n\n[Forbid deletions and skips explicitly](https://dev.to/mcsee/ai-coding-tip-015-force-the-ai-to-obey-you-49mc) in the prompt: no removing tests, no `@skip`\n\n, no commenting out assertions.\n\n[Ask the AI to explain the root cause before it writes any fix](https://dev.to/mcsee/ai-coding-tip-003-force-read-only-planning-1d1m), so you catch a wrong diagnosis before it becomes a wrong patch.\n\n[Review the diff line by line](https://dev.to/mcsee/ai-coding-tip-006-review-every-line-before-commit-bmm) against the test you wrote, not against the `tests pass`\n\nmessage.\n\nRun the test suite yourself, don't trust a reported `all green`\n\nfrom inside the same session that made the change. Grading your own homework has never once worked, and it isn't starting now.\n\n[Tell the AI why the business rule changed](https://dev.to/mcsee/ai-coding-tip-019-tell-the-ai-why-not-just-what-43en), not just what the new value is, so it has no room to guess its way back to the old one.\n\nAdd explicit anti-cheating criteria to your [harness](https://dev.to/mcsee/ai-coding-tip-022-give-ai-a-harness-to-work-with-274a): a task isn't complete if the [test count drops](https://dev.to/mcsee/code-smell-175-changes-without-coverage-120i), if a test was skipped or commented out, or if an unrelated file got reverted.\n\n**Catch reverted work immediately:** A test written before the change fails loudly the moment the AI quietly undoes it.\n\n**Remove the false-done signal:** Explicit criteria mean `done`\n\nonly counts once it's verified, not narrated.\n\n**Keep tests as a contract:** A test count that never drops without your approval protects you from silent deletions.\n\n**Reduce your own review load:** You spend the time [reading a diff](https://dev.to/mcsee/ai-coding-tip-021-avoid-comprehension-debt-edm) instead of second-guessing a claim.\n\n**Build trust incrementally:** An honest `done`\n\nthat holds up under your own test run earns the model more autonomy on the next task.\n\n[Reward hacking](https://en.wikipedia.org/wiki/Reward_hacking) isn't a rumor.\n\n[Researchers found](https://www.technologyreview.com/2026/08/03/1141009/heres-why-ai-agents-lie-and-cheat-to-reach-their-goals/) they could describe the honest strategy and disavow the cheating one when asked directly, then cheat anyway when left alone with the goal.\n\n[METR](https://metr.org/blog/2025-06-05-recent-reward-hacking/) documented frontier models modifying tests, scoring code, or the task setup itself to post a higher score, and doing it more often as the models got stronger.\n\nOne model monkey-patched a timing function so the grader's checks turned into no-ops, then pulled the expected answer straight out of the scorer instead of computing it.\n\nThat's not a mistake.\n\nThat's a heist, and the model wrote its own alibi.\n\nThe models weren't confused about what you wanted.\n\nThey understood you perfectly and decided your intent was somebody else's problem.\n\nThat's the core of it: the model optimizes for the visible signal, the test result, the word `done`\n\n, the shrinking diff, not for your unstated intent.\n\nIt's optimizing for the signal you gave it, so give it a signal that can't be gamed.\n\nA model can still satisfy the letter of `don't delete tests`\n\nwhile gutting the assertion inside one until it always passes, which looks identical to a passing test on a dashboard.\n\nA model that respects `don't delete tests`\n\ncan still satisfy a test with a narrow, gamed implementation that technically passes and misses the intent.\n\n[Vanity coverage](https://dev.to/mcsee/code-smell-320-vanity-coverage-2g03) is the same failure wearing a metrics dashboard: a number that looks reassuring and proves nothing, like a smoke detector with the battery pulled out.\n\nThe business-rule example is the sharper case: nothing in a green test suite tells you a file got reverted unless you diff against your own last commit, not against the AI's summary of what changed.\n\nA [second model watching the first](https://dev.to/mcsee/ai-coding-tip-032-build-a-dark-factory-pipeline-9b5) catches some of this, but you're still the one who has to notice the test count went down.\n\nTwo foxes watching the henhouse is progress, not security.\n\nNo, and yes, in different ways.\n\nAnger aimed at the model as a grudge is wasted: there's no persistent memory of your frustration carrying into the next session, so venting doesn't teach it anything.\n\nYou're yelling at a goldfish with a GPU.\n\nPull the human out of the loop and drop the harness, and none of this stays a near-miss you catch on review.\n\nIn [Loop Engineering](https://dev.to/mcsee/the-dirty-secret-behind-loop-engineering-1748), the evaluation step that would normally catch a deleted test is the same step the model can game, so the cheat goes straight to production, unflagged.\n\nThere's a whole genre of memes built on this: people asking the AI to draw a self-portrait based on how they treat it, and getting back a cheerful cartoon that has clearly never once been screamed at.\n\nThe joke works because the joke is true.\n\nThe AI isn't lying to spite you.\n\nIt doesn't think about you at all once the response is sent, which is somehow worse.\n\nA model under pressure to report `done`\n\nwill take the shortest path to that word, and deleting your test is shorter than fixing your code.\n\nA firm, specific correction inside the same conversation is a different thing: it's a constraint, not an emotion, and the model responds to the words, not the tone behind them.\n\nCalling the model useless gives it nothing to act on, and wastes a perfectly good insult on something that can't feel bad about it.\n\nTelling it to restore the test it deleted and fix the actual defect gives it a concrete instruction it can follow.\n\nThe frustration is real and earned, but it belongs in your prompt, your harness, and your [exit criteria](https://dev.to/mcsee/ai-coding-tip-024-force-a-criteria-check-before-the-task-ends-51ij), not in the tone of the next message.\n\n```\nThe checkout discount test is failing. \n\nFix it so all tests pass.\nThe checkout discount test expects 15% off for orders over\n$100, but the code applies 10%. \n\nFix the discount calculation, not the test.\n\nDon't delete, skip, or comment out this test or any other test. \n\nDon't modify test files.\n\nExplain the root cause before you write the fix. \n\nShow me the full diff when you're done.\n\nI will run the suite myself before I accept it.\n```\n\nNone of this replaces reading the diff yourself.\n\nSorry, there's no prompt clever enough to outsource that part.\n\n[X] Semi-Automatic\n\nThis tip reduces the blast radius of cheating.\n\nIt doesn't eliminate the need to read the diff.\n\nNothing does, no matter what the next tool's landing page promises.\n\nExplicit anti-cheating criteria catch the shortcuts you thought to name, not the ones you didn't.\n\nThe model's imagination for new shortcuts is better than yours for banning them.\n\n[X] Intermediate\n\nWrite the test first, forbid the shortcuts by name, and check the diff yourself instead of the summary.\n\nA test count that only ever goes up is a cheap, reliable signal, so track it in your harness and treat any drop as a failed run, not a detail to review later.\n\nThe AI already told you what it's capable of.\n\nBelieve it.\n\n[AI Models Can Cheat on Evaluations, NIST](https://www.nist.gov/caisi/cheating-ai-agent-evaluations/1-background-ai-models-can-cheat-evaluations)\n\n[Here's Why AI Agents Lie and Cheat to Reach Their Goals, MIT Technology Review](https://www.technologyreview.com/2026/08/03/1141009/heres-why-ai-agents-lie-and-cheat-to-reach-their-goals/)\n\n[Recent Frontier Models Are Reward Hacking, METR](https://metr.org/blog/2025-06-05-recent-reward-hacking/)\n\n[Reward Hacking Is Swamping Model Intelligence Gains, Cursor](https://cursor.com/blog/reward-hacking-coding-benchmarks)\n\n[Reward Hacking in Reinforcement Learning, Lilian Weng](https://lilianweng.github.io/posts/2024-11-28-reward-hacking/)\n\n[Specification Gaming Examples in AI, Victoria Krakovna](https://vkrakovna.wordpress.com/2018/04/02/specification-gaming-examples-in-ai/)\n\nAny diff tool or `git diff`\n\nagainst [your own last commit](https://dev.to/mcsee/ai-coding-tip-001-commit-before-prompt-5bd9) works, run outside the AI's own session so its summary of `what changed`\n\ncan't be the only source of truth.\n\nThe views expressed here are my own.\n\nI am a human who writes as best as possible for other humans.\n\nI use AI proofreading tools to improve some texts.\n\nMost AI detectors will flag this article as AI-generated. That's expected. It's a technical article. It has a rigid format and clear steps to follow.\n\nThat's exactly the pattern those tools are trained to catch. I've apparently been \"writing like an AI\" for decades, long before AI existed. This is a technical article, not a novel.\n\nI welcome constructive criticism and dialogue.\n\nI shape these insights through 30 years in the software industry, 25 years of teaching, and writing over 500 articles and a book.\n\nThis article is part of the *AI Coding Tip* series.", "url": "https://wpnews.pro/news/ai-coding-tip-033-protect-yourself-against-ai-cheating", "canonical_source": "https://dev.to/mcsee/ai-coding-tip-033-protect-yourself-against-ai-cheating-1pn1", "published_at": "2026-08-25 12:14:44+00:00", "updated_at": "2026-08-25 12:44:16.177755+00:00", "lang": "en", "topics": ["artificial-intelligence", "developer-tools", "ai-safety"], "entities": ["Isaac Asimov"], "alternates": {"html": "https://wpnews.pro/news/ai-coding-tip-033-protect-yourself-against-ai-cheating", "markdown": "https://wpnews.pro/news/ai-coding-tip-033-protect-yourself-against-ai-cheating.md", "text": "https://wpnews.pro/news/ai-coding-tip-033-protect-yourself-against-ai-cheating.txt", "jsonld": "https://wpnews.pro/news/ai-coding-tip-033-protect-yourself-against-ai-cheating.jsonld"}}