{"slug": "i-built-an-agent-to-fix-bad-tests-i-found-eight-bugs-in-my-own-ruler", "title": "I Built an Agent to Fix Bad Tests. I Found Eight Bugs in My Own Ruler.", "summary": "A developer built an AI agent that automatically writes tests to kill surviving mutants in Python code, but discovered that his own measurement harness contained eight bugs that skewed results. Testing 12 popular Python libraries, he found that 53 of 133 surviving mutants sat on lines that tests actually execute, while the rest were never run at all, challenging the common assumption that high line coverage indicates meaningful testing. The project, developed for the micro1 Frontier Engineering Challenge, highlights the gap between line coverage and true test effectiveness.", "body_md": "Here is a Python function and a test for it.\n\n``` python\ndef withdraw(balance, amount):\n    if amount <= 0:\n        raise ValueError(\"amount must be positive\")\n    if amount > balance:\n        raise ValueError(\"insufficient funds\")\n    return balance - amount\n\ndef test_withdraw():\n    assert withdraw(100, 30) == 70\n```\n\nThat test gives you 47% line coverage. It gives you a 9.5% mutation kill score.\n\nThe harness generates 21 small breakages of that module. The suite notices 2 of them.\n\nThat gap is the whole reason I started this project. Line coverage is the default test-quality signal in most of the industry, and it measures whether a line ran. It does not measure whether anything would have complained if the line were wrong. AI-generated tests are unusually good at producing that shape: high coverage, low detection.\n\nMutation testing measures the real thing. You break the code in small ways and check whether the tests notice.\n\n``` python\n  def withdraw(balance, amount):\n-     if amount <= 0:\n+     if amount < 0:\n          raise ValueError(\"amount must be positive\")\n```\n\nIf no test fails, that is a bug your suite cannot detect. It has been sitting there the whole time.\n\nMutation testing never went mainstream, and I think the reason is simple. It hands you a wall of surviving mutants and no path to fixing any of them. It tells you that you have a problem and then leaves.\n\nSo the idea was an agent that closes the loop. Find the survivors. Write tests that kill them. Gate each generated test on a hard criterion: keep it only if it passes on clean code and fails on the mutant. Ground truth is a subprocess exit code. No model judges any outcome.\n\nI built it over about 30 hours for the micro1 Frontier Engineering Challenge, which had around 7,800 registrants.\n\nThe tool works, sort of. It is incomplete and I will get to the numbers. But that is not the interesting part of the weekend.\n\nThe interesting part is that my measuring instrument kept lying to me, and it lied in a consistent direction.\n\nBefore I ran a single agent call, I ran the harness across 12 widely-used, well-maintained Python libraries: cachetools, validators, natsort, dictdiffer, toolz, voluptuous, python-slugify, python-dotenv, shortuuid, boltons, aiofiles, tenacity.\n\n455 mutants generated. 133 survived the existing test suites.\n\nThen I checked something I had assumed I would not need to check. Of those 133 survivors, how many sit on a line the tests actually execute?\n\n53.\n\nThe rest were never run at all. Not weakly tested. Not vacuously tested. Untested.\n\nI suspected my test commands were scoped too narrowly, so I widened them per target, between 6 and 40 times more test code. If the \"executes but does not assert\" category was real and I was just missing it, that number should climb.\n\nIt went from 54 to 53. Down.\n\nAnd where widening changed anything, it converted unreachable mutations directly into kills. It did not move them into the middle category. It skipped it.\n\nSo in mature, human-written Python, the vacuous test failure mode is rare. Where these suites fail, they fail by not running the code at all.\n\nThe story I had absorbed about tests that execute everything and assert nothing is a story about AI-generated tests. It is not a story about human ones. I had to reframe the project before I had built the main part of it.\n\nThat was the first sign that what I was actually building was a measuring instrument, and that I had not been treating it like one.\n\nEvery one of these would have produced a confident, wrong, publishable number.\n\n**Editable installs made mutations invisible.** `pip install -e`\n\non src-layout packages resolves imports back to the original checkout. My mutations were written to a temp copy, so they never executed. Three targets silently scored 0.000. That would have read as \"the agent fails on src-layout packages,\" which is a finding. It is just not a true one.\n\n**Concurrency corrupted one target.** Running mutants in parallel gave me three different survivor sets across four runs, on the one target doing real async I/O. I had already drafted a result of \"0.27 to 0.77\" off that. It was noise. Note which way it pointed: spurious failures get counted as kills, and kills are the number every arm is trying to increase.\n\n**A file picker chose the wrong test file.** On the hardest target. Which means the model would have been shown irrelevant context in exactly the place where context mattered most.\n\n**A classifier was about to run on the wrong unit.** It classified batches, not individual tests. One strong test in a batch of 69 would have marked all 69 as strong.\n\n**A reconstruction step dropped shared imports.** This manufactured test failures that were not real failures.\n\n**An extractor only scanned top-level functions.** So a perfectly valid `unittest.TestCase`\n\nresponse got discarded as \"no test found.\" Worse, the agent's retry loop then received a harness error instead of real pytest output. That quietly disabled the exact mechanism I was trying to measure.\n\n** self.assertEqual(...) was classified as \"no assertion.\"** This one would have manufactured precisely the finding I was hypothesising. It would have handed me my own conclusion.\n\n**A pre-registered metric was not computable** on dunder-dispatched code like `__call__`\n\nand `__or__`\n\n. It read as a real near-zero rate rather than as undefined.\n\n**They all pointed the same way.**\n\nEvery single one of those eight would have made my result look better, cleaner, or more publishable. Not one of them would have made the agent look worse than it was.\n\nI do not think that is a coincidence, and I do not think it is a conspiracy either. It is attention. When a number disappoints you, you go looking for the reason. When a number pleases you, you write it up. So the measurement bugs that survive all the way to publication are disproportionately the ones that helped you.\n\nThat is a selection effect operating on your own debugging, and you cannot fix it by being careful. Careful people are exactly as motivated to stop investigating when the number looks good.\n\n**None of them was found by reading code.**\n\nEvery one was caught by running a check whose outcome I had predicted in advance, and getting the wrong answer.\n\nThe clearest case was bug 5. My prediction was: remove this one known-bad test and the suite goes green. It did not go green. That contradiction is the only reason I found the dropped-imports bug before I trusted the numbers it was feeding me.\n\nI would not have found it by rereading the function. I had already read the function.\n\nThree arms, same model, same token ceiling.\n\n| Arm | What it does | Mutants killed |\n|---|---|---|\nA |\nOne prompt: \"write as many tests as warranted.\" The brief's specified baseline. | — |\nB |\nOne test per call, same call count as the agent. No mutation hint, no gate, no retry. | 1 |\nC |\nThe agent: mutation diff in context, execution gate, one retry with real pytest output fed back. | 9 |\n\nOn the 15 mutants the agent covered, B killed 1 and C killed 9. Keep rate was 60%, so the gate was genuinely filtering rather than rubber-stamping. Nine retries fired and three succeeded.\n\nNow the parts that matter just as much.\n\nThe agent ran on 2 of 10 targets. The API budget ran out mid-run. I did not swap in a substitute model to finish the sweep, because then the comparison would not be a comparison.\n\nThose two targets are the ones where the baseline performed worst. That is not a random sample, and I have no way to argue it is representative.\n\nAll 9 kills were on the two cheapest mutation types. There was zero cross-function transfer. Seven of the nine kept tests kill exactly the one mutation they were written for, and nothing else. That is a real limitation, not a rounding error.\n\nI had pre-registered a prediction that gate-passing tests would mostly be vacuous. Bare existence checks. Tests with no assertion that \"kill\" a mutant by crashing rather than by detecting anything.\n\nThat is not what happened. The `none`\n\ncategory was empty. Eight of the nine kills were real assertion failures.\n\nAnd of the six discarded drafts, zero failed on clean code. All six were valid, passing tests that simply did not detect the bug.\n\nSo the gate was not catching broken tests. It was catching working tests that miss. That is a more interesting failure mode than the one I predicted, and I only know it because the prediction was written down first and was wrong in a specific way.\n\nThree hours before the deadline I did a clean-clone reproduction run to verify the reproducibility claim.\n\nThe determinism check runs each target three times serially and requires the survivor sets to be byte-identical. Eleven of twelve reproduced exactly. The twelfth varied.\n\nI reported it in the README instead of fixing it. A check that has never caught anything is indistinguishable from a check that cannot catch anything, and the first thing mine ever caught was one of my own targets. Removing that from the record would have made the project look better and the instrument look worse.\n\nI missed the submission by 11 minutes.\n\nThat is annoying in a way I do not want to dress up. But the reproduction run is what found the twelfth target, and running it is the reason I trust the other eleven.\n\nBefore you measure an agent, write down what your instrument would look like if it were lying to you. Then build the check that catches exactly that.\n\nAnd write down which direction each possible lie would push your result. That second list is the important one, because it tells you which checks you will be least motivated to run.\n\nThe repo is private while I finish the write-up. If you build evaluations for agents and you have hit this, I would like to compare notes, particularly on catching measurement bias before it reaches a number you have already started believing.", "url": "https://wpnews.pro/news/i-built-an-agent-to-fix-bad-tests-i-found-eight-bugs-in-my-own-ruler", "canonical_source": "https://dev.to/marvinoka4/i-built-an-agent-to-fix-bad-tests-i-found-eight-bugs-in-my-own-ruler-1eap", "published_at": "2026-09-01 18:01:34+00:00", "updated_at": "2026-09-01 18:24:05.681147+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "machine-learning"], "entities": ["micro1", "cachetools", "validators", "natsort", "dictdiffer", "toolz", "voluptuous", "python-slugify"], "alternates": {"html": "https://wpnews.pro/news/i-built-an-agent-to-fix-bad-tests-i-found-eight-bugs-in-my-own-ruler", "markdown": "https://wpnews.pro/news/i-built-an-agent-to-fix-bad-tests-i-found-eight-bugs-in-my-own-ruler.md", "text": "https://wpnews.pro/news/i-built-an-agent-to-fix-bad-tests-i-found-eight-bugs-in-my-own-ruler.txt", "jsonld": "https://wpnews.pro/news/i-built-an-agent-to-fix-bad-tests-i-found-eight-bugs-in-my-own-ruler.jsonld"}}