{"slug": "my-tests-were-green-they-caught-0-of-10-real-bugs", "title": "My tests were green. They caught 0 of 10 real bugs.", "summary": "A developer built Saboteur, a mutation-testing tool for end-to-end browser tests that plants realistic git-patch bugs, rebuilds the app, and reruns the suite to measure which bugs slip through. On a demo shop, four starter tests caught 0 of 10 planted bugs; after Claude wrote new Kane CLI tests that had to pass on the correct app and fail on the bugged one, the suite caught 10 of 10. A red-team/blue-team duel between two Claude agents ended 3-3, with the suite growing from 18 to 21 tests.", "body_md": "I had a small web shop with four end-to-end tests. All four passed, every time. I felt good about them.\n\nThen I broke the shop on purpose, ten different ways, and ran the same four tests against each broken version. They caught **zero**.\n\nThat's the problem I built **Saboteur** to measure: a green suite tells you nothing about what it misses. I built it for the KaneCLI Online Hackathon by TestMu AI.\n\nLive report: [https://softalpha0.github.io/saboteur/](https://softalpha0.github.io/saboteur/)\n\nCode: [https://github.com/softalpha0/saboteur](https://github.com/softalpha0/saboteur)\n\nSaboteur plants one small, realistic bug at a time, builds the app, and runs your test suite against it. If every test still passes, the bug \"slipped\". For each bug that slipped, Claude writes a new test, and the test is only kept if it **passes on the correct app and fails on the bugged one**. No test gets in on the model's word alone.\n\nThis is basically mutation testing, but for end-to-end browser tests instead of unit tests, with an AI closing the gaps and a real browser acting as the judge.\n\n**The tests** are Kane CLI `_test.md` files: plain English steps that Kane runs in a real Chrome browser. Here is one of the four starter tests:\n\n```\n---\nmode: testing\nurl: http://localhost:4173\ntags: [starter, cart]\nmax_steps: 30\n---\n\n# Cart shows the added item\n\n## Add the mug\nOpen the home page and add the Ceramic Mug to the cart.\nVerify the cart badge in the header shows 1.\n\n## Open the cart\nOpen the cart page.\nVerify a line item named \"Ceramic Mug\" is listed with a unit price of $14.50.\n```\n\n**The bugs** are git patches. Each one is the kind of mistake that ships for real. This one makes the \"+\" quantity button do nothing:\n\n``` js\n-        onClick={() => onChange(value + 1)}\n+        onClick={() => onChange(value)}\n```\n\nThe ten bugs in the demo shop include a cart total that skips the last item, a discount code that takes 1% instead of 10%, a checkout that accepts the email `a@`, and a cart that empties when you refresh.\n\n**The loop** for each bug:\n\n`git apply` the patch, build it, serve it.`--no-adaptive-heal`). With healing on, Kane would try to work around a broken page, which is great for flaky selectors and terrible for catching bugs.\nThis took most of the work, and it's the part I'd tell anyone building something similar to care about most.\n\nA failed test is not the same as a caught bug. While building this I saw tests fail because:\n\n`browser_disconnected`)` analyzer_failed`)\nIf you count those as catches, your score is a lie. So Saboteur:\n\nThat second rule saved me on a real run. On two bugs, a test failed with `analyzer_failed`, which had nothing to do with the bug. The confirm run passed, the suite re-ran clean, and both bugs were correctly scored as slipped instead of fake catches.\n\nFor every bug that slipped, Claude gets the bug's description, the app's routes and facts, and the existing tests, and writes a new `_test.md`. Then Kane runs it twice:\n\n| Run | Build | Healing | Must | \n|---|---|---|---|\n| A | correct app | on | pass | \n| B | bugged app | off | fail | \n\nIf run A fails, the test is wrong about the app. If run B passes, the test doesn't catch the bug. Either way, Kane's failure text goes back to Claude for another try, up to three attempts. Only tests that pass both checks join the suite.\n\n**Demo shop:** the 4 starter tests caught **0 of 10** bugs. With Claude's Kane-proven tests: **10 of 10**.\n\n**Duel mode.** I then pointed two Claude agents at each other. Red reads the current tests and invents new bugs they would miss. Blue writes tests to catch them. Kane referees, and a bug only counts if Kane can see it in the browser (the check passes on the bugged app and fails on the correct one).\n\nFinal score: **Red 3, Blue 3.** Red found three real holes:\n\nBlue closed all three, and the suite grew from 18 to 21 tests.\n\n**An app I didn't write.** A demo app I built myself is easy to game, so I also ran it on the official React [TodoMVC](https://github.com/tastejs/todomvc) with ten new bugs. Four typical starter tests (add, complete, delete, edit a todo) caught **2 of 10**. Claude has written Kane-proven tests for 6 of the 8 misses so far. The final two were cut short when Kane lost its sign-in mid-run.\n\n`cmd` launcher, so my timeout killed the launcher and left the real process alive. The fix was killing the whole process tree, plus a watchdog that stops a run when no output arrives for ten minutes.`doctor` command now checks free space before it starts.\n\n```\ngit clone https://github.com/softalpha0/saboteur\ncd saboteur\npnpm i\nkane-cli login\npnpm saboteur doctor --replay\npnpm saboteur run --replay\n```\n\n`--replay` needs no Claude key. It uses the tests Claude already wrote and Kane already proved. To point it at your own app, edit `saboteur.config.json` (build command, serve command, URL) and add your own bug patches.\n\nThe question I keep coming back to: **how many of your tests would still pass if the feature they test was broken?** I'd love to hear what you find if you try it.", "url": "https://wpnews.pro/news/my-tests-were-green-they-caught-0-of-10-real-bugs", "canonical_source": "https://dev.to/softalpha_0/my-tests-were-green-they-caught-0-of-10-real-bugs-g9h", "published_at": "2026-11-20 02:40:49+00:00", "updated_at": "2026-11-20 02:42:57.896025+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "large-language-models", "ai-safety"], "entities": ["Saboteur", "Kane CLI", "TestMu AI", "Claude", "Anthropic", "TodoMVC", "React"], "also_reported_by": [], "takeaway": "Gate every AI-written test on a two-run check — it must pass on the correct build and fail on the bugged build — and treat browser_disconnected or analyzer_failed failures as slipped bugs, not catches; this took the demo suite from 0/10 to 10/10 bugs caught.", "recommendation": {"advice": "Before trusting an AI-generated test, run it against both the correct and the deliberately bugged build and keep it only if it passes the first and fails the second.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/my-tests-were-green-they-caught-0-of-10-real-bugs", "markdown": "https://wpnews.pro/news/my-tests-were-green-they-caught-0-of-10-real-bugs.md", "text": "https://wpnews.pro/news/my-tests-were-green-they-caught-0-of-10-real-bugs.txt", "jsonld": "https://wpnews.pro/news/my-tests-were-green-they-caught-0-of-10-real-bugs.jsonld"}}