Mutate the Grader, Not the Code: A Weekly Self-Audit for AI C++ Patch Evals

A developer has proposed a self-audit technique for AI-generated C++ patch evaluation harnesses in which seeded faulty patches, called mutants, are pushed through the real grading pipeline to identify graders that fail to reject them. The approach targets grader weaknesses rather than missing test cases, flagging exit-code-only grading, baseline-normalized output that absorbs real diagnostics, and assertions that patches can edit. The audit relies on a clean git worktree, a manifest declaring which graders must reject each mutant, and a harness that reports per-grader status as machine-readable output.

Example scenario illustrative, not a reported incident : a team's eval harness has returned PASS on every AI-generated C++ patch for three weeks. Then a patch replaces a bounded comparison with a modulo on a signed value. The unit grader stays green because no golden case feeds a negative operand. The sanitizer grader stays green because it runs the same test binary, which never reaches the branch. Nothing in the report is lying — the harness simply stopped being able to see that class of bug. Green means "the cases we wrote still pass". It does not mean "the grader can still detect this failure". Those are different properties, and only the second one decays silently. The technique below audits the second property directly: seeded faulty patches, known as mutants, are pushed through the real harness, and every grader that fails to reject them is reported as blind. No model names, quotas, or benchmark numbers are assumed; the artifact is a plain script and a manifest. Exit-code-only grading. The harness records returncode = 0 as a failure and nothing else. A patch that compiles, runs, and prints a wrong number passes. So does one that emits a new warning nobody promoted to an error. Baseline-normalized output. Warnings and diagnostics are diffed against a stored baseline file. When the baseline is regenerated after a noisy commit, the diff quietly absorbs the next real diagnostic. Assertions the patch can edit. The grader greps test output for a string, and the test file lives in the same tree the model may modify. The patch adjusts the expected string; the grader confirms the adjustment. Each of these is a grader weakness, not a case weakness. Adding golden cases will not fix any of them, which is why pruning and shuffling the case set does not surface the problem either. The audit needs three inputs: a clean git worktree, a list of small patches with known faults, and a harness entry point that can report per-grader status as machine-readable output. A manifest declares, for each mutant, which graders must reject it. mutants/manifest.yaml illustrative mutants: - id: signed-modulo-clamp diff: mutants/signed modulo clamp.diff why: replaces a bounded comparison with a modulo must fail: unit clamp, prop clamp bounds - id: unchecked-alloc-size diff: mutants/unchecked alloc size.diff why: drops a size guard before memcpy must fail: asan guard - id: silent-float-narrowing diff: mutants/silent float narrowing.diff why: narrows double to float without a cast warning gate must fail: warnings as errors The must fail list is the contract. A mutant that survives a listed grader is a finding; a mutant that fails extra graders is fine and often informative. Dirty state is the most common cause of false "invalid diff" verdicts. A dedicated worktree keeps the audit off the developer's working tree. git worktree add ../eval-audit HEAD python tools/mutant audit.py \ --manifest mutants/manifest.yaml \ --worktree ../eval-audit \ --report audit-$ date -u +%Y%m%d .md The runner below is illustrative and has not been executed as part of this article. It expects python -m evals.grade --json to print {"graders": {"<id ": {"status": "pass"|"fail"}}} on stdout. bash /usr/bin/env python3 """Illustrative grader self-audit. Not executed for this article.""" import json, pathlib, subprocess, sys import yaml ROOT = pathlib.Path file .resolve .parent.parent def sh cmd, cwd : return subprocess.run cmd, cwd=cwd, capture output=True, text=True def reset worktree : sh "git", "checkout", "--", "." , worktree sh "git", "clean", "-fdq" , worktree def apply worktree, diff path : diff path = str diff path if sh "git", "apply", "--check", diff path , worktree .returncode = 0: return False return sh "git", "apply", diff path , worktree .returncode == 0 def grade worktree : """Returns graders dict, error . Never treats a crash as a grader failure.""" p = sh sys.executable, "-m", "evals.grade", "--json" , worktree try: return json.loads p.stdout "graders" , None except json.JSONDecodeError, KeyError : return None, f"no JSON from harness exit {p.returncode} " def audit manifest path, worktree : manifest = yaml.safe load pathlib.Path manifest path .read text rows = for m in manifest "mutants" : reset worktree if not apply worktree, ROOT / m "diff" : rows.append {"id": m "id" , "verdict": "invalid-diff", "survived": } continue graders, err = grade worktree if graders is None: rows.append {"id": m "id" , "verdict": "infra-error", "survived": , "detail": err} continue survived = g for g in m "must fail" if graders.get g, {} .get "status" = "fail" rows.append { "id": m "id" , "verdict": "caught" if not survived else "blind", "survived": survived, } reset worktree return rows Three verdicts matter, and they are not interchangeable. caught is a healthy grader. blind is the signal the audit exists for. infra-error means the harness itself broke — it must never be counted as a pass, otherwise a crash in the grader loop reads as perfect detection. A single global number hides the useful part. Aggregate by grader id so a weakness localizes: python from collections import defaultdict def kill rates rows : seen, killed = defaultdict int , defaultdict int for r in rows: if r "verdict" not in "caught", "blind" : continue for g in r.get "must fail", : seen g += 1 if g not in r "survived" : killed g += 1 return {g: killed g / seen g for g in seen} A grader below roughly its own historical rate is worth a look, and a grader at zero is definitionally blind. The thresholds are a team convention; they are not a standard, and picking one without a baseline of stable weeks is guesswork. Not every survivor is a bug, and treating them all as one leads teams to abandon the audit. Record the bucket in the report. A manifest that grows without retiring equivalent mutants becomes noise within a few months. Hand-authored mutants are high quality and slow to produce. A model can draft candidates from the function under test plus the current grader list, and a human then decides which are real faults versus equivalents. The cheap part of that loop is generation, and the expensive part is review — which is exactly where the human should stay. This is also the only place MonkeyCode participates in the workflow described here. Its free model access covers the candidate-generation step, and the free server option is a place to run the nightly audit so it does not consume CI minutes or a developer's laptop. The project's outreach materials describe both options; specific quota figures and server hardware change, so read the current numbers on the official page rather than planning capacity from any blog post, this one included. Disclosure: This article was prepared as part of MonkeyCode's product outreach. A generation prompt that respects the constraint the audit needs — compiles cleanly, changes behavior — reads roughly like this: Given the function below and these existing graders list , propose 5 patches that a compile without new errors, b change observable behavior on at least one reachable input, c plausibly evade at least one listed grader. For each, state the input class that exposes the fault and which grader should catch it. Two rules keep model-generated mutants honest. Every candidate is compiled before it enters the manifest, since a patch that fails to build is not a mutant. And every candidate is reviewed for equivalence by a human, since a model that believes two behaviors differ is not evidence that they do. An audit that always prints caught is worse than no audit. Plant a known failure before trusting the report: replace one grader's body with return True in a scratch branch, run the audit, and confirm that grader's kill rate drops to zero and the run exits non-zero. in the audit entry point blind = r "id" for r in rows if r "verdict" == "blind" errors = r "id" for r in rows if r "verdict" == "infra-error" if blind or errors: print report rows, kill rates rows sys.exit 1 If the planted failure is not reported, the problem is the audit, not the graders. This self-check is the difference between a dashboard and a measurement. The audit assumes deterministic graders and a harness that exposes per-grader status. Flaky timing or network checks make survivors ambiguous, and the first fix should be determinism, not more mutants. It also assumes someone owns the manifest. A stale mutant set stops finding things while continuing to print green — the same failure mode it was built to detect. Skip this if golden cases barely exist yet, if the harness only reports pass/fail with no per-grader granularity, or if a single engineer owns both the harness and every patch under test. The cost lands on the harness owner, and the benefit only appears when grader coverage and patch volume are both large enough for silent blindness to be plausible. Graders rot quietly. A short weekly run that shoves known-bad patches through the real harness converts that decay into a number someone can act on. Teams already running a patch eval loop can start with three mutants and one grader contract; that is enough to learn whether the report is describing the model's patches or the harness's blind spots.