# Freeze a Holdout Before You Quote a Coding-Agent Score > Source: > Published: 2026-09-17 05:15:27+00:00 You are in a Friday ranking review. Someone pastes a table: Agent B hit 71 percent. Last week it was 58. The room relaxes. Then you ask whether the system prompt changed after the last failure dump. It did. Three times. The 71 is a fitted number. It is not a measurement. That is how coding-agent scoreboards rot. You inspect failures, you edit the prompt, you rerun the same tasks, you publish the new pass rate. The worksheet became the exam. If you want a number you can defend, you freeze a holdout first and you never tune against it. This article is a protocol, not a leaderboard. You will split a small task suite, lock the split, score two folders with two different commands, and refuse to publish anything that touched the tune set. Cheap inference and a spare server only matter at the end, as a place to rerun the frozen pass. They do not change the rules. A coding agent is a policy. Prompts, tools, retries, and stop conditions are all knobs. Every knob you turn on the same tasks moves the score toward the tasks. That is ordinary overfitting. It is not controversial in ML. It becomes controversial only when the slide says "the agent is better" instead of "the agent was fitted to this folder." You need three things before a percentage is a measurement: a dataset with a frozen denominator, metrics that still mean something after a prompt edit, and controls that catch a broken grader. Miss any one of those and you are quoting a demo. The protocol below is intentionally small. It fits on a laptop. It also fits on a free server if you want the lockfile to live somewhere you are not babysitting. You freeze four objects, once: You do not freeze the tune folder. You are allowed to wreck it. You may change prompts, swap tools, add retries, and read every failing diff. You may not copy a holdout task into tune because it "looks similar." You may not publish tune scores. You may not add holdout tasks after you have seen them fail. If that feels strict, good. Strict is the point. Start with tasks you can grade in isolation. Each task is a directory, not a row in a spreadsheet. Spreadsheets hide missing oracles. ``` suite/ tasks/ T001_csv_header_drift/ prompt.md repo/ tests/ test_hidden.py oracle.patch meta.json T002_retry_backoff_jitter/ ... ``` `meta.json` stays boring on purpose: ``` { "id": "T001_csv_header_drift", "language": "python", "timeout_sec": 120, "oracle_must_pass": true, "tags": ["csv", "parser"] } ``` Write every task against four rules. If a task breaks a rule, it is not in the suite yet. Do not collect 400 tasks on day one. Twenty well-formed tasks beat two hundred demos. You can grow the suite later. You cannot un-see a holdout. Use a seed, not your intuition. Intuition puts the gnarly tasks in tune so the holdout "looks fair." That is another form of fitting. ``` # split_suite.py # Proposed local tool. Run it once, then commit the manifest. from __future__ import annotations import argparse, json, random from pathlib import Path def task_ids(root: Path) -> list[str]: tasks = sorted(p.name for p in (root / "tasks").iterdir() if p.is_dir()) if len(tasks) < 12: raise SystemExit("Need at least 12 tasks before a holdout is meaningful.") return tasks def main() -> None: parser = argparse.ArgumentParser() parser.add_argument("--suite", type=Path, required=True) parser.add_argument("--seed", type=int, required=True) parser.add_argument("--holdout-frac", type=float, default=0.4) args = parser.parse_args() ids = task_ids(args.suite) rng = random.Random(args.seed) shuffled = ids[:] rng.shuffle(shuffled) cut = max(5, int(round(len(shuffled) * args.holdout_frac))) holdout = sorted(shuffled[:cut]) tune = sorted(shuffled[cut:]) overlap = set(tune) & set(holdout) if overlap: raise SystemExit(f"split leaked: {sorted(overlap)}") manifest = { "seed": args.seed, "holdout_frac": args.holdout_frac, "tune": tune, "holdout": holdout, "frozen": True, "note": "Do not edit lists after first agent run on holdout.", } out = args.suite / "split_manifest.json" if out.exists(): raise SystemExit(f"{out} already exists; refusing to resplit") out.write_text(json.dumps(manifest, indent=2) + "\n") print(f"tune={len(tune)} holdout={len(holdout)} wrote {out}") if __name__ == "__main__": main() ``` Run it like this and then treat the file as a contract: ``` python split_suite.py --suite ./suite --seed 20260917 git add suite/split_manifest.json suite/tasks git commit -m "Freeze holdout split seed=20260917" ``` If the file already exists, the script exits. That is not a bug. Resplitting after you have seen holdout failures is how 71 percent happens. Pass rate alone is a trap. It moves when you drop hard tasks, when you swallow timeouts as failures, and when hidden tests are too thin. Publish a row, not a cell. | Metric | Formula | Publish? | Why | |---|---|---|---| | Tune pass rate | passes / attempted on tune | No | You fitted the prompt here | | Holdout pass rate | passes / attempted on holdout | Yes, if frozen | This is the measurement | | Exclusion rate | excluded / scheduled | Yes | Shows infra and timeout load | | Oracle sanity | oracles that pass / tasks | Must be 1.0 | If < 1.0, stop scoring agents | | Weak-test flag | unfixed repo already passes | Must be 0 | Task does not belong in the suite | | Prompt fingerprint | hash of the quoted command | Yes | Stops silent knob changes | Attempted means the job produced a patch or an explicit empty diff. A missing interpreter is excluded, not failed. If you dump exclusions into failures, the agent that crashes less looks smarter. It is not. You still need a decision table for the meeting, because someone will ask to "just add the three tasks it almost got." | Situation | Publish the holdout number? | Reason | |---|---|---| | Frozen split, frozen prompt, oracle sanity 1.0 | Yes | Measurement | | Prompt edited after reading holdout diffs | No | Holdout is now tune | | Tasks added to holdout after first run | No | Denominator moved | | Tune pass rate only | No | Fitted | | Oracle patch fails hidden tests | No | Grader is broken | | Timeout counted as fail | No | Infra mixed into skill | Print that table above the score. If people skip it, they are not reading a benchmark. They are reading a press line. The grader should not have a `--all` flag that publishes. Make the publish path refuse the tune folder. ``` # score_split.py # Proposed grader wrapper. Wire run_task() to your existing harness. from __future__ import annotations import argparse, hashlib, json, subprocess, sys from pathlib import Path def fingerprint(argv: list[str]) -> str: blob = "\0".join(argv).encode() return hashlib.sha256(blob).hexdigest()[:16] def load_manifest(suite: Path) -> dict: path = suite / "split_manifest.json" data = json.loads(path.read_text()) if set(data["tune"]) & set(data["holdout"]): raise SystemExit("manifest overlap; split is invalid") return data def oracle_ok(task_dir: Path) -> bool: repo = task_dir / "repo" patch = task_dir / "oracle.patch" tests = task_dir / "tests" r = subprocess.run( ["python", "-m", "pytest", str(tests), "-q"], cwd=apply_patch(repo, patch), # you provide apply_patch() capture_output=True, text=True, ) return r.returncode == 0 def main() -> None: parser = argparse.ArgumentParser() parser.add_argument("--suite", type=Path, required=True) parser.add_argument("--split", choices=["tune", "holdout"], required=True) parser.add_argument("--publish", action="store_true") parser.add_argument("--agent-cmd", nargs=argparse.REMAINDER, required=True) args = parser.parse_args() if args.publish and args.split != "holdout": raise SystemExit("refusing to publish a non-holdout split") man = load_manifest(args.suite) ids = man[args.split] agent_fp = fingerprint(args.agent_cmd) rows = [] for task_id in ids: task_dir = args.suite / "tasks" / task_id if not oracle_ok(task_dir): rows.append({"id": task_id, "status": "oracle_fail"}) continue result = run_task(task_dir, args.agent_cmd) # you provide run_task() rows.append(result) oracle_fails = [r for r in rows if r["status"] == "oracle_fail"] if oracle_fails: raise SystemExit(f"grader broken on {[r['id'] for r in oracle_fails]}") excluded = [r for r in rows if r["status"] == "excluded"] attempted = [r for r in rows if r["status"] in {"pass", "fail"}] passes = [r for r in attempted if r["status"] == "pass"] report = { "split": args.split, "publishable": bool(args.publish), "agent_fingerprint": agent_fp, "scheduled": len(ids), "attempted": len(attempted), "passed": len(passes), "excluded": len(excluded), "holdout_pass_rate": (len(passes) / len(attempted)) if attempted else None, "exclusion_rate": len(excluded) / len(ids) if ids else None, "rows": rows, } out = args.suite / f"report_{args.split}_{agent_fp}.json" out.write_text(json.dumps(report, indent=2) + "\n") print(out) if not args.publish: print("not publishable (tune or unpublished holdout)") if __name__ == "__main__": main() ``` Daily loop on tune: ``` python score_split.py --suite ./suite --split tune --agent-cmd -- ./run_agent.sh ``` Quoted run, once, after you stop touching knobs: ``` python score_split.py --suite ./suite --split holdout --publish --agent-cmd -- ./run_agent.sh --prompt prompts/frozen_v3.md ``` Keep `prompts/frozen_v3.md` in the same commit as the report. If the file changes and the fingerprint does not, your wrapper is lying. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The protocol is local. You can run it with any agent command. MonkeyCode is relevant only as cheap capacity: free model access for the tune loop, and a free server option for the overnight holdout pass so the lockfile and the report sit on a box you do not have to keep open. That is the whole product role here. No model names, no quota theater, no "superhuman" claim. Use the free model access on `--split tune` while you are still reading diffs. You are not allowed to quote those scores, so burning paid capacity there is usually waste. When the prompt is frozen, send one holdout job to the free server option, write `report_holdout_*.json` next to `split_manifest.json`, and stop. If you keep iterating after that job, you no longer have a holdout. You have a second tune folder with extra latency. If MonkeyCode disappeared tomorrow, the split, the oracle check, and the publish guard would still be the method. That is the test of whether this article is a workflow or an ad. A publishable report should look dull: ``` split: holdout publishable: true agent_fingerprint: 9c2e0b1a7d44f0c1 scheduled: 8 attempted: 7 passed: 3 excluded: 1 holdout_pass_rate: 0.429 exclusion_rate: 0.125 ``` Three passes out of seven attempted is not a press release. It is a number with a denominator. The excluded task stays visible. If someone later "cleans" the suite by deleting that task, the next report is a different experiment. Say so, or do not compare the two numbers. Watch for three quiet cheats. First, prompt drift. The fingerprint must match the command you put in the slide. A retry flag is a different agent. Second, task shopping. Moving one holdout failure into tune after the fact is not data cleaning. It is peeking. Third, grader rot. If an OS update makes pytest skip a file, oracle sanity will save you. If you skip oracle sanity, you will rank agents on a suite that no longer tests what you think it tests. This protocol does not estimate human time. It does not produce confidence intervals. It does not prove that holdout tasks are drawn from your production distribution. A 40 percent holdout on 12 tasks is still a toy. It only answers one question: did you stop fitting before you quoted the number? It also cannot catch contamination you cannot see. If the hidden tests were copied from a public kata, a model may have seen them in pretraining. A freeze does not fix leakage. It only fixes your own peeking. Flaky tests will still lie. Run the oracle twice if your suite has time or network noise. If the oracle is flaky, the task is not ready. Do not use this split if you have fewer than twelve tasks. You cannot freeze a holdout you do not have. Do not use it to certify a vendor, a hire, or a production SLA. A frozen folder of synthetic bugs is not your incident stream. Do not use it if you need to change hidden tests during an incident. Fix production first. Rebuild the suite later. Mixing those jobs is how oracles rot. Do not use a free shared server if the repos under test contain secrets, customer data, or private credentials. The protocol assumes throwaway tasks. If you only want a demo that a coding agent can edit a file, skip the holdout. Say it is a demo. The damage starts when the demo grows a percentage. You can say: on this frozen holdout, with this command line, the agent passed 3 of 7 attempted tasks, excluded 1 of 8 scheduled, and every oracle still passes. You cannot say the agent is better than most developers. You cannot say it is superhuman. You cannot say last week's 58 percent is comparable unless the manifest, the tests, and the fingerprint match. That sentence is longer than 71 percent. It is also true. If you want a cheap box to run the frozen pass overnight, MonkeyCode's free server option is sufficient for this protocol. The number still comes from the holdout, not from the box.