{"slug": "freeze-a-holdout-before-you-quote-a-coding-agent-score", "title": "Freeze a Holdout Before You Quote a Coding-Agent Score", "summary": "A developer has published a protocol for scoring coding agents that requires freezing a holdout task suite before any prompt or tool tuning, arguing that pass rates published after iterating on the same tasks are fitted numbers rather than measurements. The approach splits a small directory-based task suite with a seeded script, locks the manifest, and forbids publishing any score that touched the tune set. The author notes the protocol runs on a laptop or a spare server and is meant to replace leaderboard-style demo numbers with defensible measurements.", "body_md": "You are in a Friday ranking review. Someone pastes a table: Agent B hit 71 percent. Last week it was 58. The room relaxes. Then you ask whether the system prompt changed after the last failure dump. It did. Three times. The 71 is a fitted number. It is not a measurement.\n\nThat is how coding-agent scoreboards rot. You inspect failures, you edit the prompt, you rerun the same tasks, you publish the new pass rate. The worksheet became the exam. If you want a number you can defend, you freeze a holdout first and you never tune against it.\n\nThis article is a protocol, not a leaderboard. You will split a small task suite, lock the split, score two folders with two different commands, and refuse to publish anything that touched the tune set. Cheap inference and a spare server only matter at the end, as a place to rerun the frozen pass. They do not change the rules.\n\nA coding agent is a policy. Prompts, tools, retries, and stop conditions are all knobs. Every knob you turn on the same tasks moves the score toward the tasks. That is ordinary overfitting. It is not controversial in ML. It becomes controversial only when the slide says \"the agent is better\" instead of \"the agent was fitted to this folder.\"\n\nYou need three things before a percentage is a measurement: a dataset with a frozen denominator, metrics that still mean something after a prompt edit, and controls that catch a broken grader. Miss any one of those and you are quoting a demo.\n\nThe protocol below is intentionally small. It fits on a laptop. It also fits on a free server if you want the lockfile to live somewhere you are not babysitting.\n\nYou freeze four objects, once:\n\nYou do not freeze the tune folder. You are allowed to wreck it. You may change prompts, swap tools, add retries, and read every failing diff. You may not copy a holdout task into tune because it \"looks similar.\" You may not publish tune scores. You may not add holdout tasks after you have seen them fail.\n\nIf that feels strict, good. Strict is the point.\n\nStart with tasks you can grade in isolation. Each task is a directory, not a row in a spreadsheet. Spreadsheets hide missing oracles.\n\n```\nsuite/\n  tasks/\n    T001_csv_header_drift/\n      prompt.md\n      repo/\n      tests/\n        test_hidden.py\n      oracle.patch\n      meta.json\n    T002_retry_backoff_jitter/\n      ...\n```\n\n`meta.json` stays boring on purpose:\n\n```\n{\n  \"id\": \"T001_csv_header_drift\",\n  \"language\": \"python\",\n  \"timeout_sec\": 120,\n  \"oracle_must_pass\": true,\n  \"tags\": [\"csv\", \"parser\"]\n}\n```\n\nWrite every task against four rules. If a task breaks a rule, it is not in the suite yet.\n\nDo not collect 400 tasks on day one. Twenty well-formed tasks beat two hundred demos. You can grow the suite later. You cannot un-see a holdout.\n\nUse a seed, not your intuition. Intuition puts the gnarly tasks in tune so the holdout \"looks fair.\" That is another form of fitting.\n\n```\n# split_suite.py\n# Proposed local tool. Run it once, then commit the manifest.\nfrom __future__ import annotations\n\nimport argparse, json, random\nfrom pathlib import Path\n\ndef task_ids(root: Path) -> list[str]:\n    tasks = sorted(p.name for p in (root / \"tasks\").iterdir() if p.is_dir())\n    if len(tasks) < 12:\n        raise SystemExit(\"Need at least 12 tasks before a holdout is meaningful.\")\n    return tasks\n\ndef main() -> None:\n    parser = argparse.ArgumentParser()\n    parser.add_argument(\"--suite\", type=Path, required=True)\n    parser.add_argument(\"--seed\", type=int, required=True)\n    parser.add_argument(\"--holdout-frac\", type=float, default=0.4)\n    args = parser.parse_args()\n\n    ids = task_ids(args.suite)\n    rng = random.Random(args.seed)\n    shuffled = ids[:]\n    rng.shuffle(shuffled)\n    cut = max(5, int(round(len(shuffled) * args.holdout_frac)))\n    holdout = sorted(shuffled[:cut])\n    tune = sorted(shuffled[cut:])\n    overlap = set(tune) & set(holdout)\n    if overlap:\n        raise SystemExit(f\"split leaked: {sorted(overlap)}\")\n\n    manifest = {\n        \"seed\": args.seed,\n        \"holdout_frac\": args.holdout_frac,\n        \"tune\": tune,\n        \"holdout\": holdout,\n        \"frozen\": True,\n        \"note\": \"Do not edit lists after first agent run on holdout.\",\n    }\n    out = args.suite / \"split_manifest.json\"\n    if out.exists():\n        raise SystemExit(f\"{out} already exists; refusing to resplit\")\n    out.write_text(json.dumps(manifest, indent=2) + \"\\n\")\n    print(f\"tune={len(tune)} holdout={len(holdout)} wrote {out}\")\n\nif __name__ == \"__main__\":\n    main()\n```\n\nRun it like this and then treat the file as a contract:\n\n```\npython split_suite.py --suite ./suite --seed 20260917\ngit add suite/split_manifest.json suite/tasks\ngit commit -m \"Freeze holdout split seed=20260917\"\n```\n\nIf the file already exists, the script exits. That is not a bug. Resplitting after you have seen holdout failures is how 71 percent happens.\n\nPass rate alone is a trap. It moves when you drop hard tasks, when you swallow timeouts as failures, and when hidden tests are too thin. Publish a row, not a cell.\n\n| Metric | Formula | Publish? | Why | \n|---|---|---|---|\n| Tune pass rate | passes / attempted on tune | No | You fitted the prompt here | \n| Holdout pass rate | passes / attempted on holdout | Yes, if frozen | This is the measurement | \n| Exclusion rate | excluded / scheduled | Yes | Shows infra and timeout load | \n| Oracle sanity | oracles that pass / tasks | Must be 1.0 | If < 1.0, stop scoring agents | \n| Weak-test flag | unfixed repo already passes | Must be 0 | Task does not belong in the suite | \n| Prompt fingerprint | hash of the quoted command | Yes | Stops silent knob changes | \n\nAttempted means the job produced a patch or an explicit empty diff. A missing interpreter is excluded, not failed. If you dump exclusions into failures, the agent that crashes less looks smarter. It is not.\n\nYou still need a decision table for the meeting, because someone will ask to \"just add the three tasks it almost got.\"\n\n| Situation | Publish the holdout number? | Reason | \n|---|---|---|\n| Frozen split, frozen prompt, oracle sanity 1.0 | Yes | Measurement | \n| Prompt edited after reading holdout diffs | No | Holdout is now tune | \n| Tasks added to holdout after first run | No | Denominator moved | \n| Tune pass rate only | No | Fitted | \n| Oracle patch fails hidden tests | No | Grader is broken | \n| Timeout counted as fail | No | Infra mixed into skill | \n\nPrint that table above the score. If people skip it, they are not reading a benchmark. They are reading a press line.\n\nThe grader should not have a `--all` flag that publishes. Make the publish path refuse the tune folder.\n\n```\n# score_split.py\n# Proposed grader wrapper. Wire run_task() to your existing harness.\nfrom __future__ import annotations\n\nimport argparse, hashlib, json, subprocess, sys\nfrom pathlib import Path\n\ndef fingerprint(argv: list[str]) -> str:\n    blob = \"\\0\".join(argv).encode()\n    return hashlib.sha256(blob).hexdigest()[:16]\n\ndef load_manifest(suite: Path) -> dict:\n    path = suite / \"split_manifest.json\"\n    data = json.loads(path.read_text())\n    if set(data[\"tune\"]) & set(data[\"holdout\"]):\n        raise SystemExit(\"manifest overlap; split is invalid\")\n    return data\n\ndef oracle_ok(task_dir: Path) -> bool:\n    repo = task_dir / \"repo\"\n    patch = task_dir / \"oracle.patch\"\n    tests = task_dir / \"tests\"\n    r = subprocess.run(\n        [\"python\", \"-m\", \"pytest\", str(tests), \"-q\"],\n        cwd=apply_patch(repo, patch),  # you provide apply_patch()\n        capture_output=True,\n        text=True,\n    )\n    return r.returncode == 0\n\ndef main() -> None:\n    parser = argparse.ArgumentParser()\n    parser.add_argument(\"--suite\", type=Path, required=True)\n    parser.add_argument(\"--split\", choices=[\"tune\", \"holdout\"], required=True)\n    parser.add_argument(\"--publish\", action=\"store_true\")\n    parser.add_argument(\"--agent-cmd\", nargs=argparse.REMAINDER, required=True)\n    args = parser.parse_args()\n\n    if args.publish and args.split != \"holdout\":\n        raise SystemExit(\"refusing to publish a non-holdout split\")\n\n    man = load_manifest(args.suite)\n    ids = man[args.split]\n    agent_fp = fingerprint(args.agent_cmd)\n    rows = []\n    for task_id in ids:\n        task_dir = args.suite / \"tasks\" / task_id\n        if not oracle_ok(task_dir):\n            rows.append({\"id\": task_id, \"status\": \"oracle_fail\"})\n            continue\n        result = run_task(task_dir, args.agent_cmd)  # you provide run_task()\n        rows.append(result)\n\n    oracle_fails = [r for r in rows if r[\"status\"] == \"oracle_fail\"]\n    if oracle_fails:\n        raise SystemExit(f\"grader broken on {[r['id'] for r in oracle_fails]}\")\n\n    excluded = [r for r in rows if r[\"status\"] == \"excluded\"]\n    attempted = [r for r in rows if r[\"status\"] in {\"pass\", \"fail\"}]\n    passes = [r for r in attempted if r[\"status\"] == \"pass\"]\n    report = {\n        \"split\": args.split,\n        \"publishable\": bool(args.publish),\n        \"agent_fingerprint\": agent_fp,\n        \"scheduled\": len(ids),\n        \"attempted\": len(attempted),\n        \"passed\": len(passes),\n        \"excluded\": len(excluded),\n        \"holdout_pass_rate\": (len(passes) / len(attempted)) if attempted else None,\n        \"exclusion_rate\": len(excluded) / len(ids) if ids else None,\n        \"rows\": rows,\n    }\n    out = args.suite / f\"report_{args.split}_{agent_fp}.json\"\n    out.write_text(json.dumps(report, indent=2) + \"\\n\")\n    print(out)\n    if not args.publish:\n        print(\"not publishable (tune or unpublished holdout)\")\n\nif __name__ == \"__main__\":\n    main()\n```\n\nDaily loop on tune:\n\n```\npython score_split.py --suite ./suite --split tune --agent-cmd -- ./run_agent.sh\n```\n\nQuoted run, once, after you stop touching knobs:\n\n```\npython score_split.py --suite ./suite --split holdout --publish --agent-cmd -- ./run_agent.sh --prompt prompts/frozen_v3.md\n```\n\nKeep `prompts/frozen_v3.md` in the same commit as the report. If the file changes and the fingerprint does not, your wrapper is lying.\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nThe protocol is local. You can run it with any agent command. MonkeyCode is relevant only as cheap capacity: free model access for the tune loop, and a free server option for the overnight holdout pass so the lockfile and the report sit on a box you do not have to keep open. That is the whole product role here. No model names, no quota theater, no \"superhuman\" claim.\n\nUse the free model access on `--split tune` while you are still reading diffs. You are not allowed to quote those scores, so burning paid capacity there is usually waste. When the prompt is frozen, send one holdout job to the free server option, write `report_holdout_*.json` next to `split_manifest.json`, and stop. If you keep iterating after that job, you no longer have a holdout. You have a second tune folder with extra latency.\n\nIf MonkeyCode disappeared tomorrow, the split, the oracle check, and the publish guard would still be the method. That is the test of whether this article is a workflow or an ad.\n\nA publishable report should look dull:\n\n```\nsplit: holdout\npublishable: true\nagent_fingerprint: 9c2e0b1a7d44f0c1\nscheduled: 8\nattempted: 7\npassed: 3\nexcluded: 1\nholdout_pass_rate: 0.429\nexclusion_rate: 0.125\n```\n\nThree passes out of seven attempted is not a press release. It is a number with a denominator. The excluded task stays visible. If someone later \"cleans\" the suite by deleting that task, the next report is a different experiment. Say so, or do not compare the two numbers.\n\nWatch for three quiet cheats.\n\nFirst, prompt drift. The fingerprint must match the command you put in the slide. A retry flag is a different agent.\n\nSecond, task shopping. Moving one holdout failure into tune after the fact is not data cleaning. It is peeking.\n\nThird, grader rot. If an OS update makes pytest skip a file, oracle sanity will save you. If you skip oracle sanity, you will rank agents on a suite that no longer tests what you think it tests.\n\nThis protocol does not estimate human time. It does not produce confidence intervals. It does not prove that holdout tasks are drawn from your production distribution. A 40 percent holdout on 12 tasks is still a toy. It only answers one question: did you stop fitting before you quoted the number?\n\nIt also cannot catch contamination you cannot see. If the hidden tests were copied from a public kata, a model may have seen them in pretraining. A freeze does not fix leakage. It only fixes your own peeking.\n\nFlaky tests will still lie. Run the oracle twice if your suite has time or network noise. If the oracle is flaky, the task is not ready.\n\nDo not use this split if you have fewer than twelve tasks. You cannot freeze a holdout you do not have.\n\nDo not use it to certify a vendor, a hire, or a production SLA. A frozen folder of synthetic bugs is not your incident stream.\n\nDo not use it if you need to change hidden tests during an incident. Fix production first. Rebuild the suite later. Mixing those jobs is how oracles rot.\n\nDo not use a free shared server if the repos under test contain secrets, customer data, or private credentials. The protocol assumes throwaway tasks.\n\nIf you only want a demo that a coding agent can edit a file, skip the holdout. Say it is a demo. The damage starts when the demo grows a percentage.\n\nYou can say: on this frozen holdout, with this command line, the agent passed 3 of 7 attempted tasks, excluded 1 of 8 scheduled, and every oracle still passes. You cannot say the agent is better than most developers. You cannot say it is superhuman. You cannot say last week's 58 percent is comparable unless the manifest, the tests, and the fingerprint match.\n\nThat sentence is longer than 71 percent. It is also true. If you want a cheap box to run the frozen pass overnight, MonkeyCode's free server option is sufficient for this protocol. The number still comes from the holdout, not from the box.", "url": "https://wpnews.pro/news/freeze-a-holdout-before-you-quote-a-coding-agent-score", "canonical_source": "https://dev.to/byteio_501/freeze-a-holdout-before-you-quote-a-coding-agent-score-2ooa", "published_at": "2026-09-17 05:15:27+00:00", "updated_at": "2026-09-17 05:53:21.127948+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "developer-tools", "mlops", "ai-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/freeze-a-holdout-before-you-quote-a-coding-agent-score", "markdown": "https://wpnews.pro/news/freeze-a-holdout-before-you-quote-a-coding-agent-score.md", "text": "https://wpnews.pro/news/freeze-a-holdout-before-you-quote-a-coding-agent-score.txt", "jsonld": "https://wpnews.pro/news/freeze-a-holdout-before-you-quote-a-coding-agent-score.jsonld"}}