Can you actually trust your LLM eval suite to catch a regression?

Developer egnaro9 released evalmut, an open-source mutation testing tool for LLM evaluation suites, available via pip install evalmut and on GitHub under an MIT license. The tool injects 18 provenance-gated defects into a system to check whether eval suites catch regressions, with deterministic red/green results and a nonzero exit code for empty suites. It was refined through eight rounds of cold-critique with Claude Code to avoid false positives.

Can you actually trust your LLM eval suite to catch a regression? I decided to stop guessing and apply the logic of mutation testing—usually reserved for traditional software—to AI evaluations. I built a tool called evalmut that mechanically probes for holes in your testing logic. The premise is simple: inject a known defect into the system, run your evals, and see if anything actually turns red. If the test stays green despite a known failure being present, you've found a hole in your eval suite. It's not a theoretical debate; it's a reproducible failure. For anyone wanting to harden their AI workflow, here is how the technical implementation handles this: Installation: You can get it via pip install evalmut . The CLI is designed to run directly against standard Python suite files. Mutation Operators: It uses 18 different operators. Crucially, these aren't random; they are provenance-gated, meaning every operator is based on a documented production failure or a real issue tracker bug. Deterministic Results: One of the biggest pains in prompt engineering is the "LLM judge" that changes its mind. This tool avoids that entirely. The red/green status is deterministic and reproducible. I spent a significant amount of time in an adversarial loop with Claude Code /en/tags/claude%20code/ to refine this, because a mutation tester that gives false positives is worse than having no tester at all. It went through eight rounds of cold-critique to ensure that if the tool says there is a hole, there actually is one. In fact, if you run it against a completely empty suite, it exits with a nonzero status on purpose—because a suite that checks nothing should never be reported as "hole-free." If you are building a real-world LLM agent and relying on a regression suite, I highly recommend running a deep dive into your test coverage using this method. For those who want to see the logic or the academic side of the method, the implementation is available here: https://github.com/egnaro9/evalmut The repository is MIT licensed and includes a paper in the /paper directory if you want the full methodology. It's a much more rigorous way to ensure you aren't just grading your own homework. Next Stop using negation in your prompts if you want to avoid → /en/threads/6197/ these real-world AI monetization case studies https://tanyan888.com/ , with plenty of directly applicable cases. All Replies （0） No replies yet — be the first