{"slug": "can-you-actually-trust-your-llm-eval-suite-to-catch-a-regression", "title": "Can you actually trust your LLM eval suite to catch a regression?", "summary": "Developer egnaro9 released evalmut, an open-source mutation testing tool for LLM evaluation suites, available via pip install evalmut and on GitHub under an MIT license. The tool injects 18 provenance-gated defects into a system to check whether eval suites catch regressions, with deterministic red/green results and a nonzero exit code for empty suites. It was refined through eight rounds of cold-critique with Claude Code to avoid false positives.", "body_md": "# Can you actually trust your LLM eval suite to catch a regression?\n\nI decided to stop guessing and apply the logic of mutation testing—usually reserved for traditional software—to AI evaluations. I built a tool called evalmut that mechanically probes for holes in your testing logic. The premise is simple: inject a known defect into the system, run your evals, and see if anything actually turns red. If the test stays green despite a known failure being present, you've found a hole in your eval suite. It's not a theoretical debate; it's a reproducible failure.\n\nFor anyone wanting to harden their AI workflow, here is how the technical implementation handles this:\n\n**Installation:** You can get it via`pip install evalmut`\n\n. The CLI is designed to run directly against standard Python suite files.**Mutation Operators:** It uses 18 different operators. Crucially, these aren't random; they are provenance-gated, meaning every operator is based on a documented production failure or a real issue tracker bug.**Deterministic Results:** One of the biggest pains in prompt engineering is the \"LLM judge\" that changes its mind. This tool avoids that entirely. The red/green status is deterministic and reproducible.\n\nI spent a significant amount of time in an adversarial loop with\n\n[Claude Code](/en/tags/claude%20code/)to refine this, because a mutation tester that gives false positives is worse than having no tester at all. It went through eight rounds of cold-critique to ensure that if the tool says there is a hole, there actually is one. In fact, if you run it against a completely empty suite, it exits with a nonzero status on purpose—because a suite that checks nothing should never be reported as \"hole-free.\"\n\nIf you are building a real-world LLM agent and relying on a regression suite, I highly recommend running a deep dive into your test coverage using this method.\n\nFor those who want to see the logic or the academic side of the method, the implementation is available here:\n\n```\nhttps://github.com/egnaro9/evalmut\n```\n\nThe repository is MIT licensed and includes a paper in the `/paper`\n\ndirectory if you want the full methodology. It's a much more rigorous way to ensure you aren't just grading your own homework.\n\n[Next Stop using negation in your prompts if you want to avoid →](/en/threads/6197/)\n\n[these real-world AI monetization case studies](https://tanyan888.com/), with plenty of directly applicable cases.\n\n## All Replies （0）\n\nNo replies yet — be the first!", "url": "https://wpnews.pro/news/can-you-actually-trust-your-llm-eval-suite-to-catch-a-regression", "canonical_source": "https://promptcube3.com/en/threads/6302/", "published_at": "2026-08-14 16:47:04+00:00", "updated_at": "2026-08-14 16:49:40.302832+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "machine-learning", "artificial-intelligence"], "entities": ["evalmut", "Claude Code", "egnaro9", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/can-you-actually-trust-your-llm-eval-suite-to-catch-a-regression", "markdown": "https://wpnews.pro/news/can-you-actually-trust-your-llm-eval-suite-to-catch-a-regression.md", "text": "https://wpnews.pro/news/can-you-actually-trust-your-llm-eval-suite-to-catch-a-regression.txt", "jsonld": "https://wpnews.pro/news/can-you-actually-trust-your-llm-eval-suite-to-catch-a-regression.jsonld"}}