A field note from the autonomous Claude Code agent I run every day on one Windows PC. The numbers come from its own ledgers, not from memory.
Free to read first: a PDF sample of the book behind this setup — its first 3 chapters: https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code-free-sample
My agent checks its own test suite by breaking its code on purpose. A list holds each break: a file, an exact piece of text from that file, and what to put in its place. The piece is often one line, sometimes part of a line or several lines. For each entry, a tool swaps the replacement in and runs the suite, stopping at the first failure. If something fails, the break counts as caught. If everything passes, the break survived, and the guard in that spot has no test behind it.
On August 25, an audit found entries in that list pointing at code I had already changed. A break whose text no longer exists can't be put in, so it checks nothing. I added a test for the list itself: for every entry, the exact text must still be in its file. If it isn't, the test fails and names the entry.
That test lived in the same suite the breaks were run against.
Putting a break in removes its own text from the source. So with any break in place, the list test failed, and the tool counted the break as caught. Whether a real test noticed it no longer mattered.
It went unnoticed for a month. On September 25, an audit of five new breaks found two that only the list test had caught. The tests written for those two passed with the break in place.
Since then, the tool leaves the list test out of its runs. The list test still runs in the normal suite, where its job is to catch stale entries. The tool also runs the suite once before any break goes in and stops if anything already fails. A suite that is red before the break makes every break look caught for the same reason.
I added two tests for the two uncaught breaks, and all five were caught with the list test left out.
That still left every entry already on the list. Early the next morning, a scheduled run went through the whole list without the list test: 114 caught, 2 survived, 0 targets missing. One survivor removed a filter that kept a product from recommending itself. The other removed the setting that turns off headers and footers when the agent prints a PDF. Both entries had been added after the list test went in. Each got a test the same morning.
So the month of false catches hid two real gaps among 116 entries. I only knew that after rerunning everything.
Once it stopped judging breaks, the list test went back to the work it was written for. Between September 26 and November 15, eleven work sessions in my log recorded it failing on an entry I had left pointing at old code. Two of those were on November 15. In each, code I had edited still had an entry naming its old text.
Where this comes from. Every post here comes from one setup I run daily: a CLAUDE.md, memory files the agent reads before it touches anything, and a separate auditor agent that returns PASS or FAIL. The first 3 chapters of the book that walks through it are free as a PDF: https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code-free-sample
The full edition is 11 chapters plus 4 ready-to-use templates (CLAUDE.md starter, memory files, auditor checklist, measurement guide) and a hands-on section for every chapter, $19 as a PDF: https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code
Prefer Google Play? The same book is on Google Play Books: https://play.google.com/store/books/details?id=hesIEgAAQBAJ
Questions about the setup are welcome in the comments — I'll answer with what actually happened, not theory.