# Can you actually trust your LLM eval suite to catch a regression? > Source: > Published: 2026-08-14 16:47:04+00:00 # Can you actually trust your LLM eval suite to catch a regression? I decided to stop guessing and apply the logic of mutation testing—usually reserved for traditional software—to AI evaluations. I built a tool called evalmut that mechanically probes for holes in your testing logic. The premise is simple: inject a known defect into the system, run your evals, and see if anything actually turns red. If the test stays green despite a known failure being present, you've found a hole in your eval suite. It's not a theoretical debate; it's a reproducible failure. For anyone wanting to harden their AI workflow, here is how the technical implementation handles this: **Installation:** You can get it via`pip install evalmut` . The CLI is designed to run directly against standard Python suite files.**Mutation Operators:** It uses 18 different operators. Crucially, these aren't random; they are provenance-gated, meaning every operator is based on a documented production failure or a real issue tracker bug.**Deterministic Results:** One of the biggest pains in prompt engineering is the "LLM judge" that changes its mind. This tool avoids that entirely. The red/green status is deterministic and reproducible. I spent a significant amount of time in an adversarial loop with [Claude Code](/en/tags/claude%20code/)to refine this, because a mutation tester that gives false positives is worse than having no tester at all. It went through eight rounds of cold-critique to ensure that if the tool says there is a hole, there actually is one. In fact, if you run it against a completely empty suite, it exits with a nonzero status on purpose—because a suite that checks nothing should never be reported as "hole-free." If you are building a real-world LLM agent and relying on a regression suite, I highly recommend running a deep dive into your test coverage using this method. For those who want to see the logic or the academic side of the method, the implementation is available here: ``` https://github.com/egnaro9/evalmut ``` The repository is MIT licensed and includes a paper in the `/paper` directory if you want the full methodology. It's a much more rigorous way to ensure you aren't just grading your own homework. [Next Stop using negation in your prompts if you want to avoid →](/en/threads/6197/) [these real-world AI monetization case studies](https://tanyan888.com/), with plenty of directly applicable cases. ## All Replies （0） No replies yet — be the first!