An AI model wrote 69 tests for a Python module. Every one passed. Together they caught zero of the eleven bugs I had deliberately planted in that module.
A second setup, pointed at the specific bugs rather than at the module, used 17 attempts and caught ten.
That contrast is the whole project.
A mutation-testing harness. It changes source code in small ways, flips a comparison, alters a constant, deletes a raise, then runs the existing test suite and records which changes the suite fails to notice. A change nothing catches is a fault your tests cannot detect.
Coverage tells you a line ran. This tells you whether anything would have failed if that line had been wrong. Those are very different numbers. A toy module with one happy-path test showed 47% line coverage and caught 2 of 21 mutations.
Then I pointed an agent at the mutations the tests missed. For each one it writes a single test, and that test is kept only if it both passes on clean code and fails on that specific mutation. Ground truth is a subprocess exit code. No model judges any outcome.
Three approaches, same model, same token ceiling, twelve widely used Python libraries including cachetools, toolz, tenacity and boltons. 455 mutations generated, 53 of which survive the existing tests and sit on lines those tests actually execute.
| approach | caught |
|---|---|
| one prompt, "write more tests" (556 tests) | 9 of 53 |
| one test per call, no targeting (53 tests) | 2 of 53 |
| targeted at the specific fault, with a pass/fail gate | 44 of 53 |
I hand-audited the nine it missed. Seven are provably unkillable, six of those being type annotations inside TYPE_CHECKING blocks that never execute at runtime. So 44 of 46 that could be caught at all.
Three findings I think matter more than that number.
Most undetected faults are unreached code, not weak assertions. Only 53 of 133 surviving mutations sit on a line the tests execute at all. I suspected my test commands were scoped too narrowly, so I widened every one of them by 6 to 40 times. The count went from 54 to 53. Down. In mature, human-written code, tests do not mostly execute code without checking it. They mostly do not execute it.
The gate never once rejected a broken test. Across 74 attempts, every rejected draft was a valid, passing test that simply failed to detect the fault. Not one was broken. The gate turns out to have exactly one job in practice, and it is not the job I designed it for.
The tests it writes catch the fault they were shown and almost nothing else. Zero cross-function transfer across 44 kept tests. 36 of the 44 catch exactly one mutation. That is the uncomfortable result, and it belongs next to the 44 rather than underneath it.
Building the agent took a weekend. The rest of the time went into discovering that my measuring instrument kept lying to me.
Eleven bugs in the harness itself. Editable installs that made mutations invisible. Parallel execution corrupting one target. A classifier running on the wrong unit. A bytecode cache serving stale results. An outcome bucket that had never once filled, because it matched on a string my version of pytest does not emit. I had published that empty bucket as a finding.
Two things they had in common.
Every one made the results look better, or made an absence look like evidence. That is selection rather than conspiracy. Debugging is triggered by surprise, and a pleasing result is not surprising. So the filter that removes measurement bugs gets applied unevenly, hard against results you dislike and softly against results you like. The ones that flatter you survive to publication.
Not one was found by reading the code. Every single one was caught by running a check whose outcome I had predicted in advance and getting the wrong answer.
Then I published, and three readers found three more. All of them by pointing at my checks, never at my numbers. Nobody has disputed a single result. Every correction landed on the instrument.
If you build evaluations for your own work, the harness is the part worth publishing. A result is a claim people can take or leave. An instrument is something they can attack, and the attacks are what tell you whether it works. I got more out of three comment threads than out of forty hours of my own review.
Repo, with all eleven bugs documented and every result reproducible from a clean clone: killcheck repo
The longer write-ups, if you want them: Thanks to Vinh Nguyen (@vinhnguyenthanhdn), Ahmet Özel (@ahmetozel) and Zain Dana Harper (@zaindanaharper). Three findings, three checks improved, zero numbers moved.