Three weeks ago I asked an uncomfortable question about my own LLM regression suite: if a model quietly got worse in a way I care about, would any check actually go red?
I didn't reason about it. I built a tool that answers it mechanically, the way mutation testing answers it for ordinary code: inject a known defect into the system under test, run the eval suite, and report which checks stayed green. A surviving mutation is a hole in the eval. Not an argument — a hole, with a name and a reproduction.
It's called evalmut, and it's now public:
pip install evalmut
— the CLI runs against a plain Python suite fileThe part I'd defend in a design review: the tool went through eight rounds of adversarial cold-critique before I trusted it, because a mutation tester that's wrong is worse than none — it hands out false confidence about false confidence. Early rounds found real false positives in the tool itself. By round six, tool-fault false positives on a well-formed suite reached zero and stayed there. An empty suite exits nonzero on purpose: a mutation tester must never report "no holes" on a suite that checked nothing.
I work with AI agents openly — Claude Code wrote much of this under an adversarial loop I operate; the discipline above is how I keep either of us from grading our own homework.
Repo: [github.com/egnaro9/evalmut](https://github.com/egnaro9/evalmut) (MIT, tagged v0.1-paper — there's a short paper in /paper if you want the method written up properly)
If you run an eval suite you trust: point this at it before you trust it more. I'd genuinely like to hear what survives.