{"slug": "your-eval-suite-passes-i-built-the-tool-that-checks-whether-it-checks-anything", "title": "Your eval suite passes. I built the tool that checks whether it checks anything.", "summary": "A developer has released evalmut, an open-source mutation testing tool for LLM regression suites that injects known defects to reveal which checks fail to catch regressions. The tool underwent eight rounds of adversarial critique to eliminate false positives and exits nonzero on empty suites to avoid false confidence.", "body_md": "Three weeks ago I asked an uncomfortable question about my own LLM regression suite: if a model quietly got worse in a way I care about, would any check actually go red?\n\nI didn't reason about it. I built a tool that answers it mechanically, the way mutation testing answers it for ordinary code: **inject a known defect into the system under test, run the eval suite, and report which checks stayed green.** A surviving mutation is a hole in the eval. Not an argument — a hole, with a name and a reproduction.\n\nIt's called **evalmut**, and it's now public:\n\n`pip install evalmut`\n\n— the CLI runs against a plain Python suite fileThe part I'd defend in a design review: **the tool went through eight rounds of adversarial cold-critique before I trusted it**, because a mutation tester that's wrong is worse than none — it hands out false confidence about false confidence. Early rounds found real false positives in the tool itself. By round six, tool-fault false positives on a well-formed suite reached zero and stayed there. An empty suite exits nonzero on purpose: a mutation tester must never report \"no holes\" on a suite that checked nothing.\n\nI work with AI agents openly — Claude Code wrote much of this under an adversarial loop I operate; the discipline above is how I keep either of us from grading our own homework.\n\nRepo: [github.com/egnaro9/evalmut](https://github.com/egnaro9/evalmut) (MIT, tagged v0.1-paper — there's a short paper in /paper if you want the method written up properly)\n\nIf you run an eval suite you trust: point this at it before you trust it more. I'd genuinely like to hear what survives.", "url": "https://wpnews.pro/news/your-eval-suite-passes-i-built-the-tool-that-checks-whether-it-checks-anything", "canonical_source": "https://dev.to/agentdev9/your-eval-suite-passes-i-built-the-tool-that-checks-whether-it-checks-anything-2c3f", "published_at": "2026-08-14 10:54:00+00:00", "updated_at": "2026-08-14 11:06:41.999027+00:00", "lang": "en", "topics": ["developer-tools", "large-language-models", "ai-tools"], "entities": ["evalmut", "Claude Code", "egnaro9"], "alternates": {"html": "https://wpnews.pro/news/your-eval-suite-passes-i-built-the-tool-that-checks-whether-it-checks-anything", "markdown": "https://wpnews.pro/news/your-eval-suite-passes-i-built-the-tool-that-checks-whether-it-checks-anything.md", "text": "https://wpnews.pro/news/your-eval-suite-passes-i-built-the-tool-that-checks-whether-it-checks-anything.txt", "jsonld": "https://wpnews.pro/news/your-eval-suite-passes-i-built-the-tool-that-checks-whether-it-checks-anything.jsonld"}}