{"slug": "mutate-the-grader-not-the-code-a-weekly-self-audit-for-ai-c-patch-evals", "title": "Mutate the Grader, Not the Code: A Weekly Self-Audit for AI C++ Patch Evals", "summary": "A developer has proposed a self-audit technique for AI-generated C++ patch evaluation harnesses in which seeded faulty patches, called mutants, are pushed through the real grading pipeline to identify graders that fail to reject them. The approach targets grader weaknesses rather than missing test cases, flagging exit-code-only grading, baseline-normalized output that absorbs real diagnostics, and assertions that patches can edit. The audit relies on a clean git worktree, a manifest declaring which graders must reject each mutant, and a harness that reports per-grader status as machine-readable output.", "body_md": "*Example scenario (illustrative, not a reported incident):* a team's eval harness has returned `PASS` on every AI-generated C++ patch for three weeks. Then a patch replaces a bounded comparison with a modulo on a signed value. The unit grader stays green because no golden case feeds a negative operand. The sanitizer grader stays green because it runs the same test binary, which never reaches the branch. Nothing in the report is lying — the harness simply stopped being able to see that class of bug.\n\nGreen means \"the cases we wrote still pass\". It does not mean \"the grader can still detect this failure\". Those are different properties, and only the second one decays silently.\n\nThe technique below audits the second property directly: seeded faulty patches, known as mutants, are pushed through the real harness, and every grader that fails to reject them is reported as blind. No model names, quotas, or benchmark numbers are assumed; the artifact is a plain script and a manifest.\n\n**Exit-code-only grading.** The harness records `returncode != 0` as a failure and nothing else. A patch that compiles, runs, and prints a wrong number passes. So does one that emits a new warning nobody promoted to an error.\n\n**Baseline-normalized output.** Warnings and diagnostics are diffed against a stored baseline file. When the baseline is regenerated after a noisy commit, the diff quietly absorbs the next real diagnostic.\n\n**Assertions the patch can edit.** The grader greps test output for a string, and the test file lives in the same tree the model may modify. The patch adjusts the expected string; the grader confirms the adjustment.\n\nEach of these is a *grader* weakness, not a *case* weakness. Adding golden cases will not fix any of them, which is why pruning and shuffling the case set does not surface the problem either.\n\nThe audit needs three inputs: a clean git worktree, a list of small patches with known faults, and a harness entry point that can report per-grader status as machine-readable output. A manifest declares, for each mutant, which graders must reject it.\n\n```\n# mutants/manifest.yaml (illustrative)\nmutants:\n  - id: signed-modulo-clamp\n    diff: mutants/signed_modulo_clamp.diff\n    why: replaces a bounded comparison with a modulo\n    must_fail: [unit_clamp, prop_clamp_bounds]\n  - id: unchecked-alloc-size\n    diff: mutants/unchecked_alloc_size.diff\n    why: drops a size guard before memcpy\n    must_fail: [asan_guard]\n  - id: silent-float-narrowing\n    diff: mutants/silent_float_narrowing.diff\n    why: narrows double to float without a cast warning gate\n    must_fail: [warnings_as_errors]\n```\n\nThe `must_fail` list is the contract. A mutant that survives a listed grader is a finding; a mutant that fails extra graders is fine and often informative.\n\nDirty state is the most common cause of false \"invalid diff\" verdicts. A dedicated worktree keeps the audit off the developer's working tree.\n\n```\ngit worktree add ../eval-audit HEAD\npython tools/mutant_audit.py \\\n  --manifest mutants/manifest.yaml \\\n  --worktree ../eval-audit \\\n  --report audit-$(date -u +%Y%m%d).md\n```\n\nThe runner below is illustrative and has not been executed as part of this article. It expects `python -m evals.grade --json` to print `{\"graders\": {\"<id>\": {\"status\": \"pass\"|\"fail\"}}}` on stdout.\n\n``` bash\n#!/usr/bin/env python3\n\"\"\"Illustrative grader self-audit. Not executed for this article.\"\"\"\nimport json, pathlib, subprocess, sys\nimport yaml\n\nROOT = pathlib.Path(__file__).resolve().parent.parent\n\ndef sh(cmd, cwd):\n    return subprocess.run(cmd, cwd=cwd, capture_output=True, text=True)\n\ndef reset(worktree):\n    sh([\"git\", \"checkout\", \"--\", \".\"], worktree)\n    sh([\"git\", \"clean\", \"-fdq\"], worktree)\n\ndef apply(worktree, diff_path):\n    diff_path = str(diff_path)\n    if sh([\"git\", \"apply\", \"--check\", diff_path], worktree).returncode != 0:\n        return False\n    return sh([\"git\", \"apply\", diff_path], worktree).returncode == 0\n\ndef grade(worktree):\n    \"\"\"Returns (graders_dict, error). Never treats a crash as a grader failure.\"\"\"\n    p = sh([sys.executable, \"-m\", \"evals.grade\", \"--json\"], worktree)\n    try:\n        return json.loads(p.stdout)[\"graders\"], None\n    except (json.JSONDecodeError, KeyError):\n        return None, f\"no JSON from harness (exit {p.returncode})\"\n\ndef audit(manifest_path, worktree):\n    manifest = yaml.safe_load(pathlib.Path(manifest_path).read_text())\n    rows = []\n    for m in manifest[\"mutants\"]:\n        reset(worktree)\n        if not apply(worktree, ROOT / m[\"diff\"]):\n            rows.append({\"id\": m[\"id\"], \"verdict\": \"invalid-diff\", \"survived\": []})\n            continue\n        graders, err = grade(worktree)\n        if graders is None:\n            rows.append({\"id\": m[\"id\"], \"verdict\": \"infra-error\", \"survived\": [], \"detail\": err})\n            continue\n        survived = [g for g in m[\"must_fail\"]\n                    if graders.get(g, {}).get(\"status\") != \"fail\"]\n        rows.append({\n            \"id\": m[\"id\"],\n            \"verdict\": \"caught\" if not survived else \"blind\",\n            \"survived\": survived,\n        })\n    reset(worktree)\n    return rows\n```\n\nThree verdicts matter, and they are not interchangeable. `caught` is a healthy grader. `blind` is the signal the audit exists for. `infra-error` means the harness itself broke — it must never be counted as a pass, otherwise a crash in the grader loop reads as perfect detection.\n\nA single global number hides the useful part. Aggregate by grader id so a weakness localizes:\n\n``` python\nfrom collections import defaultdict\n\ndef kill_rates(rows):\n    seen, killed = defaultdict(int), defaultdict(int)\n    for r in rows:\n        if r[\"verdict\"] not in (\"caught\", \"blind\"):\n            continue\n        for g in r.get(\"must_fail\", []):\n            seen[g] += 1\n            if g not in r[\"survived\"]:\n                killed[g] += 1\n    return {g: killed[g] / seen[g] for g in seen}\n```\n\nA grader below roughly its own historical rate is worth a look, and a grader at zero is definitionally blind. The thresholds are a team convention; they are not a standard, and picking one without a baseline of stable weeks is guesswork.\n\nNot every survivor is a bug, and treating them all as one leads teams to abandon the audit.\n\nRecord the bucket in the report. A manifest that grows without retiring equivalent mutants becomes noise within a few months.\n\nHand-authored mutants are high quality and slow to produce. A model can draft candidates from the function under test plus the current grader list, and a human then decides which are real faults versus equivalents. The cheap part of that loop is generation, and the expensive part is review — which is exactly where the human should stay.\n\nThis is also the only place MonkeyCode participates in the workflow described here. Its free model access covers the candidate-generation step, and the free server option is a place to run the nightly audit so it does not consume CI minutes or a developer's laptop. The project's outreach materials describe both options; specific quota figures and server hardware change, so read the current numbers on the official page rather than planning capacity from any blog post, this one included.\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nA generation prompt that respects the constraint the audit needs — *compiles cleanly, changes behavior* — reads roughly like this:\n\n```\nGiven the function below and these existing graders [list], propose 5 patches that\n(a) compile without new errors, (b) change observable behavior on at least one\nreachable input, (c) plausibly evade at least one listed grader. For each, state\nthe input class that exposes the fault and which grader should catch it.\n```\n\nTwo rules keep model-generated mutants honest. Every candidate is compiled before it enters the manifest, since a patch that fails to build is not a mutant. And every candidate is reviewed for equivalence by a human, since a model that believes two behaviors differ is not evidence that they do.\n\nAn audit that always prints `caught` is worse than no audit. Plant a known failure before trusting the report: replace one grader's body with `return True` in a scratch branch, run the audit, and confirm that grader's kill rate drops to zero and the run exits non-zero.\n\n```\n# in the audit entry point\nblind = [r[\"id\"] for r in rows if r[\"verdict\"] == \"blind\"]\nerrors = [r[\"id\"] for r in rows if r[\"verdict\"] == \"infra-error\"]\nif blind or errors:\n    print_report(rows, kill_rates(rows))\n    sys.exit(1)\n```\n\nIf the planted failure is not reported, the problem is the audit, not the graders. This self-check is the difference between a dashboard and a measurement.\n\nThe audit assumes deterministic graders and a harness that exposes per-grader status. Flaky timing or network checks make survivors ambiguous, and the first fix should be determinism, not more mutants.\n\nIt also assumes someone owns the manifest. A stale mutant set stops finding things while continuing to print green — the same failure mode it was built to detect.\n\nSkip this if golden cases barely exist yet, if the harness only reports pass/fail with no per-grader granularity, or if a single engineer owns both the harness and every patch under test. The cost lands on the harness owner, and the benefit only appears when grader coverage and patch volume are both large enough for silent blindness to be plausible.\n\nGraders rot quietly. A short weekly run that shoves known-bad patches through the real harness converts that decay into a number someone can act on. Teams already running a patch eval loop can start with three mutants and one grader contract; that is enough to learn whether the report is describing the model's patches or the harness's blind spots.", "url": "https://wpnews.pro/news/mutate-the-grader-not-the-code-a-weekly-self-audit-for-ai-c-patch-evals", "canonical_source": "https://dev.to/devcpp_268/mutate-the-grader-not-the-code-a-weekly-self-audit-for-ai-c-patch-evals-29k4", "published_at": "2026-11-09 20:44:36+00:00", "updated_at": "2026-11-09 20:53:41.526394+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-agents", "mlops"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/mutate-the-grader-not-the-code-a-weekly-self-audit-for-ai-c-patch-evals", "markdown": "https://wpnews.pro/news/mutate-the-grader-not-the-code-a-weekly-self-audit-for-ai-c-patch-evals.md", "text": "https://wpnews.pro/news/mutate-the-grader-not-the-code-a-weekly-self-audit-for-ai-c-patch-evals.txt", "jsonld": "https://wpnews.pro/news/mutate-the-grader-not-the-code-a-weekly-self-audit-for-ai-c-patch-evals.jsonld"}}