{"slug": "coverage-theatre-your-ai-hit-90-coverage-and-still-shipped-the-bug", "title": "Coverage theatre: your AI hit 90% coverage and still shipped the bug", "summary": "A developer published a free, self-contained lab demonstrating that AI-generated test suites can reach high code coverage while still passing against deliberately broken code. In the lab, the AI's own suite passes 3/3 against a pricing function seeded with four defects, while an independent verification suite fails 8/12 and catches all four, with a mutation gate confirming the weak suite never goes red on broken code. The author argues coverage measures which lines ran, not whether a test would fail if the code were wrong, and recommends mutation testing as the cheapest check.", "body_md": "Your AI assistant just generated tests until the coverage bar turned green. 92%. Ship it?\n\nHere's the trap: coverage measures which lines *ran*, not whether a test would notice when they're *wrong*. AI made that gap free — you can generate 90% coverage in a minute, all of it happy-path, none of it discriminating. That's coverage theatre: it looks like safety, it measures activity, it proves almost nothing.\n\nA line is \"covered\" the moment a test executes it. But executing a line and asserting the *right thing* about it are different claims. An AI-generated test that calls `applyDiscount(order)` and asserts `typeof result === 'number'` covers the function — and would stay green if the discount math were completely wrong.\n\nCoverage answers \"did this code run in a test?\" The question that earns trust is \"would a test fail if this code were wrong?\" Those are not the same, and AI output routinely nails the first while skipping the second.\n\nIf a suite can't tell correct behavior from a real defect, its coverage number is decoration. The cheapest check is mutation: change `>` to `>=`, flip a boolean, swap two lines — then see if anything goes red. If the suite stays green on broken code, it isn't protecting you.\n\nI put the smallest runnable version of this in a free lab (no install, Node 20+):\n\n```\nnode run-lab.js all\n```\n\nIt hands the AI's own suite a pricing function with four seeded defects, then runs an independent verification suite against the same broken code:\n\n```\nWEAK TESTS (the AI's suite) .... PASS 3/3\nSTRONG VERIFICATION ........... FAIL 8/12   <- catches all 4 defects\nMUTATION GATE ................. PASS 4/4\nLAB GATE: PASS\n```\n\nThe AI's tests pass against code that is wrong. Verification is what fails — and that failure is the signal coverage never gave you.\n\nCoverage isn't useless — low coverage is a real red flag. But high coverage is not evidence of anything on its own, and AI made it trivially easy to manufacture. Treat the green bar as a starting question, not an answer.\n\nFree, self-contained lab: [https://github.com/chernevnikolay86-wq/ai-test-verification-lab](https://github.com/chernevnikolay86-wq/ai-test-verification-lab)\n\nI turned the full method into a book + runnable kit — the operating loop, governance, and labs across unit, web/API, instrument-cluster and CAN/UDS domains.\n\nHonest caveat: the lab systems are simulators with *declared* defects — they prove the method catches faults, not that any product is certified. That's the point: don't trust things that merely look right.", "url": "https://wpnews.pro/news/coverage-theatre-your-ai-hit-90-coverage-and-still-shipped-the-bug", "canonical_source": "https://dev.to/nikolay_chernev_a67479931/coverage-theatre-your-ai-hit-90-coverage-and-still-shipped-the-bug-5bml", "published_at": "2026-09-30 09:07:33+00:00", "updated_at": "2026-09-30 09:18:04.597107+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "artificial-intelligence"], "entities": ["GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/coverage-theatre-your-ai-hit-90-coverage-and-still-shipped-the-bug", "markdown": "https://wpnews.pro/news/coverage-theatre-your-ai-hit-90-coverage-and-still-shipped-the-bug.md", "text": "https://wpnews.pro/news/coverage-theatre-your-ai-hit-90-coverage-and-still-shipped-the-bug.txt", "jsonld": "https://wpnews.pro/news/coverage-theatre-your-ai-hit-90-coverage-and-still-shipped-the-bug.jsonld"}}