Freeze a Holdout Before You Quote a Coding-Agent Score

A developer has published a protocol for scoring coding agents that requires freezing a holdout task suite before any prompt or tool tuning, arguing that pass rates published after iterating on the same tasks are fitted numbers rather than measurements. The approach splits a small directory-based task suite with a seeded script, locks the manifest, and forbids publishing any score that touched the tune set. The author notes the protocol runs on a laptop or a spare server and is meant to replace leaderboard-style demo numbers with defensible measurements.

You are in a Friday ranking review. Someone pastes a table: Agent B hit 71 percent. Last week it was 58. The room relaxes. Then you ask whether the system prompt changed after the last failure dump. It did. Three times. The 71 is a fitted number. It is not a measurement. That is how coding-agent scoreboards rot. You inspect failures, you edit the prompt, you rerun the same tasks, you publish the new pass rate. The worksheet became the exam. If you want a number you can defend, you freeze a holdout first and you never tune against it. This article is a protocol, not a leaderboard. You will split a small task suite, lock the split, score two folders with two different commands, and refuse to publish anything that touched the tune set. Cheap inference and a spare server only matter at the end, as a place to rerun the frozen pass. They do not change the rules. A coding agent is a policy. Prompts, tools, retries, and stop conditions are all knobs. Every knob you turn on the same tasks moves the score toward the tasks. That is ordinary overfitting. It is not controversial in ML. It becomes controversial only when the slide says "the agent is better" instead of "the agent was fitted to this folder." You need three things before a percentage is a measurement: a dataset with a frozen denominator, metrics that still mean something after a prompt edit, and controls that catch a broken grader. Miss any one of those and you are quoting a demo. The protocol below is intentionally small. It fits on a laptop. It also fits on a free server if you want the lockfile to live somewhere you are not babysitting. You freeze four objects, once: You do not freeze the tune folder. You are allowed to wreck it. You may change prompts, swap tools, add retries, and read every failing diff. You may not copy a holdout task into tune because it "looks similar." You may not publish tune scores. You may not add holdout tasks after you have seen them fail. If that feels strict, good. Strict is the point. Start with tasks you can grade in isolation. Each task is a directory, not a row in a spreadsheet. Spreadsheets hide missing oracles. suite/ tasks/ T001 csv header drift/ prompt.md repo/ tests/ test hidden.py oracle.patch meta.json T002 retry backoff jitter/ ... meta.json stays boring on purpose: { "id": "T001 csv header drift", "language": "python", "timeout sec": 120, "oracle must pass": true, "tags": "csv", "parser" } Write every task against four rules. If a task breaks a rule, it is not in the suite yet. Do not collect 400 tasks on day one. Twenty well-formed tasks beat two hundred demos. You can grow the suite later. You cannot un-see a holdout. Use a seed, not your intuition. Intuition puts the gnarly tasks in tune so the holdout "looks fair." That is another form of fitting. split suite.py Proposed local tool. Run it once, then commit the manifest. from future import annotations import argparse, json, random from pathlib import Path def task ids root: Path - list str : tasks = sorted p.name for p in root / "tasks" .iterdir if p.is dir if len tasks < 12: raise SystemExit "Need at least 12 tasks before a holdout is meaningful." return tasks def main - None: parser = argparse.ArgumentParser parser.add argument "--suite", type=Path, required=True parser.add argument "--seed", type=int, required=True parser.add argument "--holdout-frac", type=float, default=0.4 args = parser.parse args ids = task ids args.suite rng = random.Random args.seed shuffled = ids : rng.shuffle shuffled cut = max 5, int round len shuffled args.holdout frac holdout = sorted shuffled :cut tune = sorted shuffled cut: overlap = set tune & set holdout if overlap: raise SystemExit f"split leaked: {sorted overlap }" manifest = { "seed": args.seed, "holdout frac": args.holdout frac, "tune": tune, "holdout": holdout, "frozen": True, "note": "Do not edit lists after first agent run on holdout.", } out = args.suite / "split manifest.json" if out.exists : raise SystemExit f"{out} already exists; refusing to resplit" out.write text json.dumps manifest, indent=2 + "\n" print f"tune={len tune } holdout={len holdout } wrote {out}" if name == " main ": main Run it like this and then treat the file as a contract: python split suite.py --suite ./suite --seed 20260917 git add suite/split manifest.json suite/tasks git commit -m "Freeze holdout split seed=20260917" If the file already exists, the script exits. That is not a bug. Resplitting after you have seen holdout failures is how 71 percent happens. Pass rate alone is a trap. It moves when you drop hard tasks, when you swallow timeouts as failures, and when hidden tests are too thin. Publish a row, not a cell. | Metric | Formula | Publish? | Why | |---|---|---|---| | Tune pass rate | passes / attempted on tune | No | You fitted the prompt here | | Holdout pass rate | passes / attempted on holdout | Yes, if frozen | This is the measurement | | Exclusion rate | excluded / scheduled | Yes | Shows infra and timeout load | | Oracle sanity | oracles that pass / tasks | Must be 1.0 | If < 1.0, stop scoring agents | | Weak-test flag | unfixed repo already passes | Must be 0 | Task does not belong in the suite | | Prompt fingerprint | hash of the quoted command | Yes | Stops silent knob changes | Attempted means the job produced a patch or an explicit empty diff. A missing interpreter is excluded, not failed. If you dump exclusions into failures, the agent that crashes less looks smarter. It is not. You still need a decision table for the meeting, because someone will ask to "just add the three tasks it almost got." | Situation | Publish the holdout number? | Reason | |---|---|---| | Frozen split, frozen prompt, oracle sanity 1.0 | Yes | Measurement | | Prompt edited after reading holdout diffs | No | Holdout is now tune | | Tasks added to holdout after first run | No | Denominator moved | | Tune pass rate only | No | Fitted | | Oracle patch fails hidden tests | No | Grader is broken | | Timeout counted as fail | No | Infra mixed into skill | Print that table above the score. If people skip it, they are not reading a benchmark. They are reading a press line. The grader should not have a --all flag that publishes. Make the publish path refuse the tune folder. score split.py Proposed grader wrapper. Wire run task to your existing harness. from future import annotations import argparse, hashlib, json, subprocess, sys from pathlib import Path def fingerprint argv: list str - str: blob = "\0".join argv .encode return hashlib.sha256 blob .hexdigest :16 def load manifest suite: Path - dict: path = suite / "split manifest.json" data = json.loads path.read text if set data "tune" & set data "holdout" : raise SystemExit "manifest overlap; split is invalid" return data def oracle ok task dir: Path - bool: repo = task dir / "repo" patch = task dir / "oracle.patch" tests = task dir / "tests" r = subprocess.run "python", "-m", "pytest", str tests , "-q" , cwd=apply patch repo, patch , you provide apply patch capture output=True, text=True, return r.returncode == 0 def main - None: parser = argparse.ArgumentParser parser.add argument "--suite", type=Path, required=True parser.add argument "--split", choices= "tune", "holdout" , required=True parser.add argument "--publish", action="store true" parser.add argument "--agent-cmd", nargs=argparse.REMAINDER, required=True args = parser.parse args if args.publish and args.split = "holdout": raise SystemExit "refusing to publish a non-holdout split" man = load manifest args.suite ids = man args.split agent fp = fingerprint args.agent cmd rows = for task id in ids: task dir = args.suite / "tasks" / task id if not oracle ok task dir : rows.append {"id": task id, "status": "oracle fail"} continue result = run task task dir, args.agent cmd you provide run task rows.append result oracle fails = r for r in rows if r "status" == "oracle fail" if oracle fails: raise SystemExit f"grader broken on { r 'id' for r in oracle fails }" excluded = r for r in rows if r "status" == "excluded" attempted = r for r in rows if r "status" in {"pass", "fail"} passes = r for r in attempted if r "status" == "pass" report = { "split": args.split, "publishable": bool args.publish , "agent fingerprint": agent fp, "scheduled": len ids , "attempted": len attempted , "passed": len passes , "excluded": len excluded , "holdout pass rate": len passes / len attempted if attempted else None, "exclusion rate": len excluded / len ids if ids else None, "rows": rows, } out = args.suite / f"report {args.split} {agent fp}.json" out.write text json.dumps report, indent=2 + "\n" print out if not args.publish: print "not publishable tune or unpublished holdout " if name == " main ": main Daily loop on tune: python score split.py --suite ./suite --split tune --agent-cmd -- ./run agent.sh Quoted run, once, after you stop touching knobs: python score split.py --suite ./suite --split holdout --publish --agent-cmd -- ./run agent.sh --prompt prompts/frozen v3.md Keep prompts/frozen v3.md in the same commit as the report. If the file changes and the fingerprint does not, your wrapper is lying. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The protocol is local. You can run it with any agent command. MonkeyCode is relevant only as cheap capacity: free model access for the tune loop, and a free server option for the overnight holdout pass so the lockfile and the report sit on a box you do not have to keep open. That is the whole product role here. No model names, no quota theater, no "superhuman" claim. Use the free model access on --split tune while you are still reading diffs. You are not allowed to quote those scores, so burning paid capacity there is usually waste. When the prompt is frozen, send one holdout job to the free server option, write report holdout .json next to split manifest.json , and stop. If you keep iterating after that job, you no longer have a holdout. You have a second tune folder with extra latency. If MonkeyCode disappeared tomorrow, the split, the oracle check, and the publish guard would still be the method. That is the test of whether this article is a workflow or an ad. A publishable report should look dull: split: holdout publishable: true agent fingerprint: 9c2e0b1a7d44f0c1 scheduled: 8 attempted: 7 passed: 3 excluded: 1 holdout pass rate: 0.429 exclusion rate: 0.125 Three passes out of seven attempted is not a press release. It is a number with a denominator. The excluded task stays visible. If someone later "cleans" the suite by deleting that task, the next report is a different experiment. Say so, or do not compare the two numbers. Watch for three quiet cheats. First, prompt drift. The fingerprint must match the command you put in the slide. A retry flag is a different agent. Second, task shopping. Moving one holdout failure into tune after the fact is not data cleaning. It is peeking. Third, grader rot. If an OS update makes pytest skip a file, oracle sanity will save you. If you skip oracle sanity, you will rank agents on a suite that no longer tests what you think it tests. This protocol does not estimate human time. It does not produce confidence intervals. It does not prove that holdout tasks are drawn from your production distribution. A 40 percent holdout on 12 tasks is still a toy. It only answers one question: did you stop fitting before you quoted the number? It also cannot catch contamination you cannot see. If the hidden tests were copied from a public kata, a model may have seen them in pretraining. A freeze does not fix leakage. It only fixes your own peeking. Flaky tests will still lie. Run the oracle twice if your suite has time or network noise. If the oracle is flaky, the task is not ready. Do not use this split if you have fewer than twelve tasks. You cannot freeze a holdout you do not have. Do not use it to certify a vendor, a hire, or a production SLA. A frozen folder of synthetic bugs is not your incident stream. Do not use it if you need to change hidden tests during an incident. Fix production first. Rebuild the suite later. Mixing those jobs is how oracles rot. Do not use a free shared server if the repos under test contain secrets, customer data, or private credentials. The protocol assumes throwaway tasks. If you only want a demo that a coding agent can edit a file, skip the holdout. Say it is a demo. The damage starts when the demo grows a percentage. You can say: on this frozen holdout, with this command line, the agent passed 3 of 7 attempted tasks, excluded 1 of 8 scheduled, and every oracle still passes. You cannot say the agent is better than most developers. You cannot say it is superhuman. You cannot say last week's 58 percent is comparable unless the manifest, the tests, and the fingerprint match. That sentence is longer than 71 percent. It is also true. If you want a cheap box to run the frozen pass overnight, MonkeyCode's free server option is sufficient for this protocol. The number still comes from the holdout, not from the box.