# Holdout Ledgers Keep Agent Scores Honest > Source: > Published: 2026-09-13 20:46:27+00:00 A headline agent percentage is not a measurement until the task ledger, metrics, and controls can be re-run by a stranger. Teams that skip that freeze often publish marketing numbers dressed as evaluation, then discover they cannot reproduce the run. This note records a compact methodology that treats the dataset as a versioned artifact rather than a mood. The scripts below are labeled examples for a local harness, not results claimed from a private benchmark farm. The temptation is familiar because leaderboards reward a single number more than they reward a durable paper trail. A coding agent that appears to solve most tasks may have seen near-duplicates, burned unbounded retries, or shared context with its own grader. An honest ranking therefore starts with provenance written before the first model call, not with a product name. The sections that follow walk through dataset construction, a metric vector, and isolation controls that keep those figures from sliding into advertisement. A usable coding dataset is a sequence of immutable task records with hashes, licenses, and difficulty labels assigned before any agent runs. Each record should name the source repository, the commit that defined the failing test, and the oracle that will later judge a patch. If that metadata is written after scores exist, the ledger is already contaminated by the ranking it pretends to justify. The analogy is a sealed evidence bag in a hallway, because once the tape is broken the later story no longer holds. Holdout is not a folder named test created on the same afternoon as the leaderboard announcement. Tasks that share a parent commit, a near-identical failing assertion, or a copied docstring should be grouped so leakage stays visible. Difficulty labels belong on the ledger before agents run, because labeling after seeing failures turns the scale into a story about the winner. A stranger should be able to check the ledger hash and rebuild the sandboxes without asking the original authors for a private archive. The JSON Lines record below is a proposal for that freeze, and it is not an executed public contest. Operators should reject extra fields that appear only after a candidate has already been scored on the same tasks. Cluster identifiers exist so near-duplicate bugs cannot be counted as independent victories during a later ranking. Licenses travel with the record because a score built on uncleared code is not a public measurement at all. ``` {"task_id":"ledger-0042","repo":"https://example.com/org/widget.git","base_sha":"8f3c1a9e","failing_tests":["tests/test_queue.py::test_drain"],"failing_assertion":"assert queue.drain() == []","oracle_sha256":"c4f19a22","license":"MIT","difficulty":"B","holdout":true,"cluster_id":"queue-drain","created_at":"2026-09-14T00:00:00Z"} ``` A tiny Python gate can refuse to score any run whose ledger digest does not match the pinned value. That refusal is the entire point of the freeze, because a mutable file is not a dataset in this method. Operators can store the digest in the same commit as the harness and block later task additions. ``` python # proposal: refuse scoring when the ledger is not pinned import hashlib, json, pathlib, sys def ledger_digest(path: str) -> str: raw = pathlib.Path(path).read_bytes() return hashlib.sha256(raw).hexdigest() def assert_pinned(expected: str, path: str = "tasks.jsonl") -> None: digest = ledger_digest(path) if digest != expected: sys.stderr.write(f"ledger mismatch: {digest}\n") raise SystemExit(2) if __name__ == "__main__": assert_pinned(sys.argv[1]) print("ledger pinned; scoring may proceed") ``` After the file is hashed, a labeled git sequence can freeze the ledger before any model token is spent. Pull requests that add tasks after a ranking exists should fail that gate rather than negotiate a new percentage. The commands below are an example of that freeze, not a claim about any contest already run. ``` python pin_ledger.py "$(sha256sum tasks.jsonl | awk '{print $1}')" git add tasks.jsonl pin_ledger.py git commit -m "freeze task ledger before agent runs" ``` A single pass rate is a trophy, and trophies travel farther than the footnotes that made them possible. A re-runnable score should carry pass rate, spent tokens, wall-clock seconds, retry count, and the harness commit beside every agent name. Without those companions, two agents can share a percentage while one quietly spent an afternoon the other was denied. The metric vector is the practical difference between a lab notebook and a brightly printed billboard. Controls belong in the same table as the candidate, or the candidate is competing against an unnamed ghost. A cheap frozen baseline run on the identical ledger shows whether the environment itself drifted between Tuesday and Thursday. Pairwise movement against that baseline matters more than absolute percentages, because the grader and the sandbox are shared. If the baseline moves by itself, the ranking is measuring the laboratory rather than measuring the agent. ``` # proposal: emit a score vector instead of a single percentage from dataclasses import dataclass, asdict import json, time @dataclass class ScoreVector: agent_id: str ledger_sha256: str harness_sha: str passed: int total: int tokens: int wall_seconds: float retries: int baseline_id: str delta_pass: float def finalize(agent_id, ledger, harness, passed, total, tokens, t0, retries, baseline_pass): vec = ScoreVector( agent_id=agent_id, ledger_sha256=ledger, harness_sha=harness, passed=passed, total=total, tokens=tokens, wall_seconds=time.time() - t0, retries=retries, baseline_id="control-frozen", delta_pass=(passed / total) - baseline_pass, ) print(json.dumps(asdict(vec), indent=2)) ``` Publishing that JSON next to a blog claim is what keeps the number from becoming marketing copy. Readers can reject the ranking when the ledger hash, the harness commit, or the baseline identifier is absent. They can also reject it when wall-clock is missing while an unbounded retry loop is left undescribed in the write-up. Silence in those fields is a result, and it should grade the evaluation as incomplete rather than praise the agent. The grader must not share a context window, a working directory, or a network namespace with the agent that produced the patch. Otherwise the oracle becomes a collaborator, and the percentage measures a conversation rather than a concrete repair. A practical control is to run the agent in one sandbox, copy only the patch artifact, and grade in a second process. That split is dull plumbing, which is exactly why glossy write-ups skip it in favor of a single percentage. A second isolation axis is the machine that hosts the harness, because authoring laptops leak helper files into eval trees. Running evaluations on the same workstation that authors the tasks invites extra scripts, cached credentials, and accidental fixtures. A dedicated evaluation host, even a modest free server, keeps the development tree from leaking into the published score. Cheap frozen models, when available as a free control stratum, belong on that host so the baseline cannot load local plugins. MonkeyCode is relevant here only as one place that currently offers free model access and a free server option for running such a separated harness. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those two availability claims are the only product facts used in this workflow, and they are not a substitute for ledger hashes. Teams that already own an isolated runner can apply the same method without that product, which is the intended test of remaining usefulness. ``` # proposal: agent writes a patch, grader sees only the artifact python run_agent.py --task-id ledger-0042 --out /tmp/patch.diff scp /tmp/patch.diff eval-host:/eval/incoming/ledger-0042.diff ssh eval-host 'python grade_patch.py --ledger tasks.jsonl --task ledger-0042 --patch /eval/incoming/ledger-0042.diff' ``` The grader should hash the incoming diff, apply it on a clean worktree at the recorded base revision, and execute only listed tests. It should not fetch extra context from the agent transcript, and it should not retry tests with mutated interpreter flags. If the patch fails to apply, the result is a failed task, not an invitation to edit the frozen ledger. That severity is what prevents the dataset from becoming a living document of the winner's private taste. Near-duplicate clustering should happen before the first agent call, using the failing assertion text rather than later model output. The proposal script below groups task identifiers that would otherwise inflate a pass rate through copied bugs. Clusters can keep one task in the reported holdout and move siblings to a leakage watch list that never enters the headline rate. Doing this after seeing scores would be another way to write the dataset around the winner, so the script belongs in the freeze commit. ``` # proposal: cluster tasks by failing assertion text before any agent run import hashlib, json, pathlib, re from collections import defaultdict def normalize(text: str) -> str: return re.sub(r"\s+", " ", text.strip().lower()) def cluster_file(path: str) -> dict: groups = defaultdict(list) for line in pathlib.Path(path).read_text().splitlines(): rec = json.loads(line) key = normalize(rec["failing_assertion"]) digest = hashlib.sha256(key.encode()).hexdigest()[:12] groups[digest].append(rec["task_id"]) return dict(groups) if __name__ == "__main__": clusters = cluster_file("tasks.jsonl") for digest, task_ids in clusters.items(): if len(task_ids) > 1: print(digest, ",".join(task_ids)) ``` Marketing numbers omit the freeze, the vector, and the isolation because those details make the story slower to read. A methodology that publishes the ledger digest, the harness commit, the baseline identifier, and the spent budget cannot hide extra retries. The same methodology also cannot claim permanence, because a ledger that never grows will eventually become stale against real repositories. Honesty here is a process of re-freezing on a calendar, not a promise that one percentage will remain interesting. This workflow does not certify models for regulated domains, and it does not replace human review on security-sensitive patches. It assumes operators can pin git commits, hash files, and keep the grader offline from the agent under test. Small demos often cannot meet that bar, and deadline pressure often collapses the rule against labeling difficulty after failures. Readers who need a vendor bake-off with contractual service promises should hire an independent lab rather than adopt this notebook. Authors who cannot publish the task sources or licenses should not present a percentage as a public ranking of agents. Teams that must mutate tests after seeing agent patches are iterating on a product, not measuring one in public. Anyone hoping free model access or a free server option will remain unchanged should treat those options as convenience rather than constants. Convenience can host the harness for a weekend experiment, yet it cannot replace the frozen ledger or the score vector. A reasonable next step is to freeze a dozen licensed tasks, run one cheap control model and one candidate, and publish both hashes. Operators who want a separated host for that control run may look at MonkeyCode's free server option after the ledger already exists. The ranking should still make sense if that product line is removed from the paragraph, because the measurement lives in the freeze.