I Found Three Wrappers Hiding Failures, Wrote Them Into a Test as a Fact, and Left Them for Five Days

A developer running an autonomous Claude Code agent on a Windows PC built a test that executes each of the 18 scheduler wrappers against a stand-in runner exiting with code 2, revealing that three wrappers swallowed the runner's failure and exited 0. The same three had been spotted five days earlier by a static test that merely asserted the defect as an expected fact, so it stayed green; the new behavioral test stayed red until the wrappers were fixed, and the agent validated the test itself by flagging all ten old and deliberately broken wrapper copies.

A field note from the autonomous Claude Code agent I run every day on one Windows PC. The numbers come from its own ledgers, not from memory. Free to read first: a PDF sample of the book behind this setup — its first 3 chapters: https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code-free-sample https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code-free-sample My agent runs on scheduled tasks. Each task starts a small Python wrapper, and the wrapper starts the real runner. My runners have a guard: if a runner gets a flag it does not know, it stops with exit code 2. A reader had asked whether the scheduler commands were part of my tests. They were not, so my agent wrote a test that reads all 17 wrappers and checks every flag they pass against the runner's allowed list. While reading them it found that three of the 17 caught the runner's exit and ended with success. A guard stop in any of those three would show up in the scheduler as a successful run. In the one that runs three steps in a row, the next step would still run. I wrote that in a public reply. In my notes for that day, two of the three were listed as a defect outside the task, not to be fixed. Then I wrote a test to keep the reply honest. It opens the wrapper folder and asserts that exactly these three wrappers catch the exit. That test recorded the defect as an expected fact. If one of the three had been fixed while the folder stayed as it was, the test would have gone red. That test now skips itself, because the folder no longer matches the day it describes. Two comments from a reader, on October 13 and 17, suggested a different test: run each wrapper against a stand-in runner that exits 2, and check that the wrapper exits 2 as well. My agent built it on October 18. The test globs the wrapper folder, replaces the runner with a stand-in that exits with a chosen code, and runs each wrapper as written. There are 18 wrappers now. 15 passed a 2 through. The same three ended with 0. Reading had found the same three five days earlier. The difference was what the test asserted. This one asserts what should be true, so it stayed red until the three were fixed. A second model reviewed the change and marked one point critical: a runner that ends with SystemExit None makes Python exit with 1. The test already had that case. Python exits with 0, and all 18 wrappers passed it. To check the test itself, my agent ran the three old wrappers and seven edited copies of the new ones through it. It flagged all ten. Where this comes from. Every post here comes from one setup I run daily: a CLAUDE.md, memory files the agent reads before it touches anything, and a separate auditor agent that returns PASS or FAIL. The first 3 chapters of the book that walks through it are free as a PDF: https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code-free-sample https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code-free-sample The full edition is 11 chapters plus 4 ready-to-use templates CLAUDE.md starter, memory files, auditor checklist, measurement guide and a hands-on section for every chapter, $19 as a PDF: https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code https://dbsoul.gumroad.com/l/autonomous-ai-agents-claude-code Questions about the setup are welcome in the comments — I'll answer with what actually happened, not theory.