Example scenario (illustrative, not a reported incident): a team's eval harness has returned PASS on every AI-generated C++ patch for three weeks. Then a patch replaces a bounded comparison with a modulo on a signed value. The unit grader stays green because no golden case feeds a negative operand. The sanitizer grader stays green because it runs the same test binary, which never reaches the branch. Nothing in the report is lying — the harness simply stopped being able to see that class of bug.
Green means "the cases we wrote still pass". It does not mean "the grader can still detect this failure". Those are different properties, and only the second one decays silently.
The technique below audits the second property directly: seeded faulty patches, known as mutants, are pushed through the real harness, and every grader that fails to reject them is reported as blind. No model names, quotas, or benchmark numbers are assumed; the artifact is a plain script and a manifest.
Exit-code-only grading. The harness records returncode != 0 as a failure and nothing else. A patch that compiles, runs, and prints a wrong number passes. So does one that emits a new warning nobody promoted to an error.
Baseline-normalized output. Warnings and diagnostics are diffed against a stored baseline file. When the baseline is regenerated after a noisy commit, the diff quietly absorbs the next real diagnostic.
Assertions the patch can edit. The grader greps test output for a string, and the test file lives in the same tree the model may modify. The patch adjusts the expected string; the grader confirms the adjustment.
Each of these is a grader weakness, not a case weakness. Adding golden cases will not fix any of them, which is why pruning and shuffling the case set does not surface the problem either.
The audit needs three inputs: a clean git worktree, a list of small patches with known faults, and a harness entry point that can report per-grader status as machine-readable output. A manifest declares, for each mutant, which graders must reject it.
mutants:
- id: signed-modulo-clamp
diff: mutants/signed_modulo_clamp.diff
why: replaces a bounded comparison with a modulo
must_fail: [unit_clamp, prop_clamp_bounds]
- id: unchecked-alloc-size
diff: mutants/unchecked_alloc_size.diff
why: drops a size guard before memcpy
must_fail: [asan_guard]
- id: silent-float-narrowing
diff: mutants/silent_float_narrowing.diff
why: narrows double to float without a cast warning gate
must_fail: [warnings_as_errors]
The must_fail list is the contract. A mutant that survives a listed grader is a finding; a mutant that fails extra graders is fine and often informative.
Dirty state is the most common cause of false "invalid diff" verdicts. A dedicated worktree keeps the audit off the developer's working tree.
git worktree add ../eval-audit HEAD
python tools/mutant_audit.py \
--manifest mutants/manifest.yaml \
--worktree ../eval-audit \
--report audit-$(date -u +%Y%m%d).md
The runner below is illustrative and has not been executed as part of this article. It expects python -m evals.grade --json to print {"graders": {"<id>": {"status": "pass"|"fail"}}} on stdout.
#!/usr/bin/env python3
"""Illustrative grader self-audit. Not executed for this article."""
import json, pathlib, subprocess, sys
import yaml
ROOT = pathlib.Path(__file__).resolve().parent.parent
def sh(cmd, cwd):
return subprocess.run(cmd, cwd=cwd, capture_output=True, text=True)
def reset(worktree):
sh(["git", "checkout", "--", "."], worktree)
sh(["git", "clean", "-fdq"], worktree)
def apply(worktree, diff_path):
diff_path = str(diff_path)
if sh(["git", "apply", "--check", diff_path], worktree).returncode != 0:
return False
return sh(["git", "apply", diff_path], worktree).returncode == 0
def grade(worktree):
"""Returns (graders_dict, error). Never treats a crash as a grader failure."""
p = sh([sys.executable, "-m", "evals.grade", "--json"], worktree)
try:
return json.loads(p.stdout)["graders"], None
except (json.JSONDecodeError, KeyError):
return None, f"no JSON from harness (exit {p.returncode})"
def audit(manifest_path, worktree):
manifest = yaml.safe_load(pathlib.Path(manifest_path).read_text())
rows = []
for m in manifest["mutants"]:
reset(worktree)
if not apply(worktree, ROOT / m["diff"]):
rows.append({"id": m["id"], "verdict": "invalid-diff", "survived": []})
continue
graders, err = grade(worktree)
if graders is None:
rows.append({"id": m["id"], "verdict": "infra-error", "survived": [], "detail": err})
continue
survived = [g for g in m["must_fail"]
if graders.get(g, {}).get("status") != "fail"]
rows.append({
"id": m["id"],
"verdict": "caught" if not survived else "blind",
"survived": survived,
})
reset(worktree)
return rows
Three verdicts matter, and they are not interchangeable. caught is a healthy grader. blind is the signal the audit exists for. infra-error means the harness itself broke — it must never be counted as a pass, otherwise a crash in the grader loop reads as perfect detection.
A single global number hides the useful part. Aggregate by grader id so a weakness localizes:
from collections import defaultdict
def kill_rates(rows):
seen, killed = defaultdict(int), defaultdict(int)
for r in rows:
if r["verdict"] not in ("caught", "blind"):
continue
for g in r.get("must_fail", []):
seen[g] += 1
if g not in r["survived"]:
killed[g] += 1
return {g: killed[g] / seen[g] for g in seen}
A grader below roughly its own historical rate is worth a look, and a grader at zero is definitionally blind. The thresholds are a team convention; they are not a standard, and picking one without a baseline of stable weeks is guesswork.
Not every survivor is a bug, and treating them all as one leads teams to abandon the audit.
Record the bucket in the report. A manifest that grows without retiring equivalent mutants becomes noise within a few months.
Hand-authored mutants are high quality and slow to produce. A model can draft candidates from the function under test plus the current grader list, and a human then decides which are real faults versus equivalents. The cheap part of that loop is generation, and the expensive part is review — which is exactly where the human should stay.
This is also the only place MonkeyCode participates in the workflow described here. Its free model access covers the candidate-generation step, and the free server option is a place to run the nightly audit so it does not consume CI minutes or a developer's laptop. The project's outreach materials describe both options; specific quota figures and server hardware change, so read the current numbers on the official page rather than planning capacity from any blog post, this one included.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
A generation prompt that respects the constraint the audit needs — compiles cleanly, changes behavior — reads roughly like this:
Given the function below and these existing graders [list], propose 5 patches that
(a) compile without new errors, (b) change observable behavior on at least one
reachable input, (c) plausibly evade at least one listed grader. For each, state
the input class that exposes the fault and which grader should catch it.
Two rules keep model-generated mutants honest. Every candidate is compiled before it enters the manifest, since a patch that fails to build is not a mutant. And every candidate is reviewed for equivalence by a human, since a model that believes two behaviors differ is not evidence that they do.
An audit that always prints caught is worse than no audit. Plant a known failure before trusting the report: replace one grader's body with return True in a scratch branch, run the audit, and confirm that grader's kill rate drops to zero and the run exits non-zero.
blind = [r["id"] for r in rows if r["verdict"] == "blind"]
errors = [r["id"] for r in rows if r["verdict"] == "infra-error"]
if blind or errors:
print_report(rows, kill_rates(rows))
sys.exit(1)
If the planted failure is not reported, the problem is the audit, not the graders. This self-check is the difference between a dashboard and a measurement.
The audit assumes deterministic graders and a harness that exposes per-grader status. Flaky timing or network checks make survivors ambiguous, and the first fix should be determinism, not more mutants.
It also assumes someone owns the manifest. A stale mutant set stops finding things while continuing to print green — the same failure mode it was built to detect.
Skip this if golden cases barely exist yet, if the harness only reports pass/fail with no per-grader granularity, or if a single engineer owns both the harness and every patch under test. The cost lands on the harness owner, and the benefit only appears when grader coverage and patch volume are both large enough for silent blindness to be plausible.
Graders rot quietly. A short weekly run that shoves known-bad patches through the real harness converts that decay into a number someone can act on. Teams already running a patch eval loop can start with three mutants and one grader contract; that is enough to learn whether the report is describing the model's patches or the harness's blind spots.