# Your mutation score was 0.81 before the agent wrote a test > Source: > Published: 2026-10-19 22:04:09+00:00 If you are building evals for coding agents, at some point the question stops being "did it change the code" and becomes "did it write the tests". Coverage is the obvious metric and the obviously wrong one: a test that calls a function without asserting anything still counts as coverage. Mutation testing is the honest replacement — mutate the code, see whether the tests notice — and on a small module it is now cheap enough to run per diff, seconds to a minute. So the metric becomes a mutation score: mutants killed over mutants tried. That is a ratio, and what it is a ratio of is the whole question. I spent a morning reading a Python repo's mutation tooling: a script that generates mutants with cosmic-ray, runs each one against the tests that coverage says touch the changed lines, and reports what survived. On one 144-line module the authors had published the breakdown of a single full run: | | count | |---|---| | mutants | 342 | | killed | 237 | | survived | 48 | | survived (string-only) | 8 | | crashed | 49 | Run that through the score they were adding — killed over tried, with crashes excluded because no test has to notice a crash: ``` score = 237 / (237 + 48 + 8) = 0.809 ``` Nobody had written a test yet. That 0.809 is the existing suite, measured on mutants drawn from that module. Now suppose you use that ratio to grade an agent: give it a change with tests to write, run the mutation score on the diff, record the number. Two failures hide in the arithmetic. **The kills are not attributable.** The runner executes every test that coverage credits with the changed lines — including the ones already in the file. So on a change inside a well-covered area, most of the kills belong to tests that predate the agent, and the score is high whatever the agent wrote. On a change in a thin spot, the same quality of new test scores low. Two diffs of the same size can differ by more than the effect you are trying to measure, and the difference is a property of the module, not of the agent. **The function that prints the ratio is usually the one that cannot attribute.** Mutation runners are fastest when they stop at the first failing test: short circuit, record "killed", move on. That is exactly what you want for the human-facing question ("do these tests catch this break?") and exactly what you lose for the eval question ("which test caught it?"). In the tool I read, the fast path passes `-x` to pytest, so a mutant is recorded as killed with no record of the killer — while a second function in the same file, the one that builds the full kill matrix, runs without `-x` and stores the sorted list of failing tests per mutant. Both numbers exist. The ratio was computed from the one that throws the attribution away. The eval number you want is the difference between two runs over the same mutant set: The difference — mutants no old test kills that a new test does — is immune to how well covered the module was before, and it is what "the agent wrote tests for this change" actually claims. A delta of zero then reads correctly as "the existing tests already killed these", where the raw ratio renders it as a 0.8 pass. It costs one more run of the same matrix, not a new instrument. If the runner already stores killers per mutant, the subtraction is a set difference; if it only stores outcomes, one run over the union of old and new tests and one over the old tests alone gets you there. A ratio shipped without its reference set is a number with two causes. The reference is almost always one more run of the same instrument, and what it buys is the ability to say whose contribution the difference is. This is not specific to mutation testing. "The agent fixed 7 of 10 issues" — against what baseline? "The prompt change cut latency by 40%" — from which distribution? The reading you can act on is the one that moves only when the thing you changed moves. If you take one thing from the number at the top: before the score means anything, compute it on the code as it was. 0.809 was the starting line, and every agent on that module started from it.