{"slug": "the-test-guarding-my-list-of-deliberate-bugs-made-every-bug-look-caught-for-a", "title": "The Test Guarding My List of Deliberate Bugs Made Every Bug Look Caught for a Month", "summary": "A developer running a daily autonomous Claude Code agent on a Windows PC found that a self-check test guarding the agent's list of deliberately injected bugs had been masking real gaps for a month. Because the list-integrity test lived in the same suite the injected bugs were run against, every injected break removed the text that test checked for, causing it to fail and the tool to count each break as caught regardless of whether a real test noticed it. After excluding the list test from mutation runs and adding a pre-run baseline check, a full rerun of 116 entries found 114 caught and 2 genuinely survived, both of which received new tests.", "body_md": "*A field note from the autonomous Claude Code agent I run every day on one Windows PC. The numbers come from its own ledgers, not from memory.*\n\n**Free to read first:** a PDF sample of the book behind this setup — its first 3 chapters: [https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code-free-sample](https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code-free-sample)\n\nMy agent checks its own test suite by breaking its code on purpose. A list holds each break: a file, an exact piece of text from that file, and what to put in its place. The piece is often one line, sometimes part of a line or several lines. For each entry, a tool swaps the replacement in and runs the suite, stopping at the first failure. If something fails, the break counts as caught. If everything passes, the break survived, and the guard in that spot has no test behind it.\n\nOn August 25, an audit found entries in that list pointing at code I had already changed. A break whose text no longer exists can't be put in, so it checks nothing. I added a test for the list itself: for every entry, the exact text must still be in its file. If it isn't, the test fails and names the entry.\n\nThat test lived in the same suite the breaks were run against.\n\nPutting a break in removes its own text from the source. So with any break in place, the list test failed, and the tool counted the break as caught. Whether a real test noticed it no longer mattered.\n\nIt went unnoticed for a month. On September 25, an audit of five new breaks found two that only the list test had caught. The tests written for those two passed with the break in place.\n\nSince then, the tool leaves the list test out of its runs. The list test still runs in the normal suite, where its job is to catch stale entries. The tool also runs the suite once before any break goes in and stops if anything already fails. A suite that is red before the break makes every break look caught for the same reason.\n\nI added two tests for the two uncaught breaks, and all five were caught with the list test left out.\n\nThat still left every entry already on the list. Early the next morning, a scheduled run went through the whole list without the list test: 114 caught, 2 survived, 0 targets missing. One survivor removed a filter that kept a product from recommending itself. The other removed the setting that turns off headers and footers when the agent prints a PDF. Both entries had been added after the list test went in. Each got a test the same morning.\n\nSo the month of false catches hid two real gaps among 116 entries. I only knew that after rerunning everything.\n\nOnce it stopped judging breaks, the list test went back to the work it was written for. Between September 26 and November 15, eleven work sessions in my log recorded it failing on an entry I had left pointing at old code. Two of those were on November 15. In each, code I had edited still had an entry naming its old text.\n\n**Where this comes from.** Every post here comes from one setup I run daily: a CLAUDE.md, memory files the agent reads before it touches anything, and a separate auditor agent that returns PASS or FAIL. The first 3 chapters of the book that walks through it are free as a PDF: [https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code-free-sample](https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code-free-sample)\n\nThe full edition is 11 chapters plus 4 ready-to-use templates (CLAUDE.md starter, memory files, auditor checklist, measurement guide) and a hands-on section for every chapter, $19 as a PDF: [https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code](https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code)\n\nPrefer Google Play? The same book is on Google Play Books: [https://play.google.com/store/books/details?id=hesIEgAAQBAJ](https://play.google.com/store/books/details?id=hesIEgAAQBAJ)\n\nQuestions about the setup are welcome in the comments — I'll answer with what actually happened, not theory.", "url": "https://wpnews.pro/news/the-test-guarding-my-list-of-deliberate-bugs-made-every-bug-look-caught-for-a", "canonical_source": "https://dev.to/dbsoul/the-test-guarding-my-list-of-deliberate-bugs-made-every-bug-look-caught-for-a-month-287j", "published_at": "2026-11-19 14:10:03+00:00", "updated_at": "2026-11-19 14:13:36.818570+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools"], "entities": ["Claude Code", "Anthropic"], "also_reported_by": [], "takeaway": "A mutation-testing guard that shares a suite with the mutation run can produce false catches: excluding the list-integrity test and adding a pre-run baseline check revealed 2 real survivors among 116 injected bugs.", "recommendation": {"advice": "Exclude self-referential integrity tests from mutation runs and fail fast if the suite is already red before injecting a break, then rerun the full list to find masked survivors.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/the-test-guarding-my-list-of-deliberate-bugs-made-every-bug-look-caught-for-a", "markdown": "https://wpnews.pro/news/the-test-guarding-my-list-of-deliberate-bugs-made-every-bug-look-caught-for-a.md", "text": "https://wpnews.pro/news/the-test-guarding-my-list-of-deliberate-bugs-made-every-bug-look-caught-for-a.txt", "jsonld": "https://wpnews.pro/news/the-test-guarding-my-list-of-deliberate-bugs-made-every-bug-look-caught-for-a.jsonld"}}