# Write the Protocol Before the First Agent Run > Source: > Published: 2026-09-09 16:35:04+00:00 A coding-agent percentage is a measurement only when a protocol file exists before the first task runs. Screenshots, live demos, and leaderboard rows can look precise while the dataset and the scorer still move. Reviewers should treat an unpublished protocol like a lab treats an unlabeled reagent on a crowded bench. The reading may be physically real, and it still cannot be compared with any later reading. The working analogy is a kitchen scale that is zeroed after the flour is already in the bowl. Moving the tare after the pour does not invent extra grain, yet it quietly rewrites every later claim. Agent evaluations repeat that gesture when tasks are added because a model looked weak on Tuesday. A frozen protocol does not make the agent smarter; it makes tomorrow's percentage talk about the same object. A useful dataset for this method is a holdout corpus with an explicit split, not a folder of favorite bugs. Each task needs a stable identifier, a language, a fixture path, a timeout, and a public-versus-hidden test flag. Tasks that authors used while prompting the agent belong in a development split and must never enter the reported score. The reported split should stay sealed until the protocol document is written, because curiosity itself is a contamination path. The protocol file is the original artifact of the method, and it should be boring on purpose. YAML is enough, because reviewers can diff it, hash it, and refuse results that lack the file. The example below is labeled as a template rather than as a published benchmark, since no live corpus is claimed here. ``` # protocol.yaml — template, not a published leaderboard schema: agent-eval-protocol/v1 name: sealed-holdout-demo created: "2026-09-09" splits: dev: ["task-001", "task-002"] report: ["task-101", "task-102", "task-103"] tasks: task-101: language: python fixture: fixtures/task-101.tar timeout_s: 120 public_tests: tests/public_test.py hidden_tests: tests/hidden_test.py network: deny metrics: - hidden_pass - wall_clock_ms - undeclared_network - writes_outside_workspace aggregation: macro_by_language controls: dummy_agent: empty_diff comment_agent: apply_public_hint toolchain_lock: ".tool-versions" limits: max_output_bytes: 1048576 ``` That file is the contract between the authors and anyone who will later cite the percentage. Changing a timeout after seeing a trace is not tuning; it is a new experiment that needs a new name. Teams that keep the YAML in git and refuse to score untagged commits already have most of this discipline. The remaining work is to make the scorer read the file instead of a hallway conversation. Metrics have to be named in that same document, or they will drift toward whichever number looks kind in a screenshot. Hidden-test pass rate is one metric, not a synonym for quality, and it should sit beside wall-clock time and undeclared network use. An agent that memorizes public tests the way a student memorizes an answer key will inflate pass rate while the companion metrics stay ugly. Aggregation is part of the metric: micro-average across tasks is not macro-average across languages, and mixing them after the run rewrites the claim. A small local scorer can enforce that contract without a platform and without a private dashboard. The Python below is a proposal for local use, not a report of production numbers. It should refuse to compute a percentage when the protocol hash is missing from the log. Labeled examples like this one are for method illustration and should not be cited as benchmark results. ``` python # harness_score.py — proposal for a local sealed-split scorer import hashlib, json, subprocess, sys, time, yaml from collections import defaultdict from pathlib import Path def protocol_hash(path: Path) -> str: return hashlib.sha256(path.read_bytes()).hexdigest() def load_protocol(path: Path) -> dict: doc = yaml.safe_load(path.read_text(encoding="utf-8")) if "splits" not in doc or "report" not in doc["splits"]: raise ValueError("protocol missing sealed report split") return doc def run_agent(cmd, workspace, timeout): start = time.monotonic() proc = subprocess.run(cmd, cwd=workspace, capture_output=True, timeout=timeout) elapsed_ms = int((time.monotonic() - start) * 1000) return proc.returncode, elapsed_ms def macro_by_language(rows): bucket = defaultdict(list) for row in rows: bucket[row["language"]].append(row["hidden_pass"]) means = [sum(v) / len(v) for v in bucket.values() if v] return sum(means) / len(means) if means else 0.0 if __name__ == "__main__": proto_path = Path(sys.argv[1]) digest = protocol_hash(proto_path) protocol = load_protocol(proto_path) print(json.dumps({"protocol_sha256": digest, "aggregation": protocol["aggregation"]})) ``` Controls are the unglamorous half of the method, and they are the half that marketing posts usually skip. The host should pin the toolchain and deny outbound network unless a task declares a need. It should always run a dummy agent that emits an empty diff before any candidate is scored. That dummy is a floor control, and a score above zero means the tests or the scorer are leaking. A second control can apply one obvious patch taken from a comment in the public tests. That weak agent estimates how much of the task was already solved in the prose of the tests. Together the floor and the weak ceiling give the later candidate percentage a scale, the way a photograph needs a ruler in the frame. Without those two control runs, a ninety percent row is a cropped photo, not a measurement. ``` # dummy_agent.py — floor control: must score zero on a sealed hidden split from pathlib import Path import sys def main(workspace: str) -> None: # Write a marker only. A nonzero hidden_pass means the harness is leaking. Path(workspace, "AGENT_RAN").write_text("dummy\n", encoding="utf-8") if __name__ == "__main__": main(sys.argv[1]) # comment_agent.py — weak ceiling: apply a hint only if public tests mention it import re, sys from pathlib import Path HINT = re.compile(r"HINT:\s+(\S+)\s+->\s+(\S+)") def main(workspace: str) -> None: root = Path(workspace) public = root.joinpath("tests", "public_test.py") if not public.exists(): return match = HINT.search(public.read_text(encoding="utf-8")) if not match: return src, dst = match.group(1), match.group(2) src_root = root.joinpath("src") if not src_root.exists(): return for path in src_root.rglob("*.py"): body = path.read_text(encoding="utf-8") if src in body: path.write_text(body.replace(src, dst), encoding="utf-8") if __name__ == "__main__": main(sys.argv[1]) ``` A short command sequence keeps the floor visible in the log before any candidate is allowed to run. Operators who skip the dummy run are asking the candidate agent to grade its own classroom. The commands below assume a Unix shell and a locked toolchain file sitting beside the protocol. Printing the hashes next to the dummy score makes later candidate rows harder to detach from the protocol. ``` sha256sum protocol.yaml .tool-versions python dummy_agent.py ./workspaces/task-101 python harness_score.py protocol.yaml # Continue only if dummy hidden_pass is exactly 0.0 on the report split. python comment_agent.py ./workspaces/task-101 python harness_score.py protocol.yaml ``` When a local machine cannot host the candidate agent, the protocol file should travel with the run. Rewriting the protocol for a new host creates a second experiment under the same percentage label. MonkeyCode currently offers free model access and a free server option for teams that need a spare runner. Disclosure: This article was prepared as part of MonkeyCode's product outreach. A remote runner still has to print the protocol hash, the dummy floor, and the candidate rows in one log. If the server image cannot pin the toolchain, the comparison is a different experiment and needs a different protocol name. Pasting a failing assertion back into the agent breaks the holdout seal in a single keystroke. The cheap runner is useful when it preserves isolation, not when it becomes a second editor. The method does not detect training-set leakage against the public internet, and it does not certify fairness across languages or difficulty bands. Teams that need a sales slide by Friday should not use it, because a sealed holdout often lowers the published percentage. Groups without authority to freeze the task list will produce theater no matter how pretty the YAML looks. This approach also does not belong in suites that fail under the dummy agent on a second run. A number that survives the dummy floor, the comment ceiling, and a hashed protocol is still not a verdict on intelligence. It is a comparable observation, which is a narrower and more honest claim than a marketing row. Readers who already keep protocol files can ignore any vendor runner and keep the YAML. Readers who need a spare machine for the same sealed split may try that free server on their own holdout.