# Your eval suite passes. I built the tool that checks whether it checks anything.

> Source: <https://dev.to/agentdev9/your-eval-suite-passes-i-built-the-tool-that-checks-whether-it-checks-anything-2c3f>
> Published: 2026-08-14 10:54:00+00:00

Three weeks ago I asked an uncomfortable question about my own LLM regression suite: if a model quietly got worse in a way I care about, would any check actually go red?

I didn't reason about it. I built a tool that answers it mechanically, the way mutation testing answers it for ordinary code: **inject a known defect into the system under test, run the eval suite, and report which checks stayed green.** A surviving mutation is a hole in the eval. Not an argument — a hole, with a name and a reproduction.

It's called **evalmut**, and it's now public:

`pip install evalmut`

— the CLI runs against a plain Python suite fileThe part I'd defend in a design review: **the tool went through eight rounds of adversarial cold-critique before I trusted it**, because a mutation tester that's wrong is worse than none — it hands out false confidence about false confidence. Early rounds found real false positives in the tool itself. By round six, tool-fault false positives on a well-formed suite reached zero and stayed there. An empty suite exits nonzero on purpose: a mutation tester must never report "no holes" on a suite that checked nothing.

I work with AI agents openly — Claude Code wrote much of this under an adversarial loop I operate; the discipline above is how I keep either of us from grading our own homework.

Repo: [github.com/egnaro9/evalmut](https://github.com/egnaro9/evalmut) (MIT, tagged v0.1-paper — there's a short paper in /paper if you want the method written up properly)

If you run an eval suite you trust: point this at it before you trust it more. I'd genuinely like to hear what survives.