Write the Protocol Before the First Agent Run

A developer argues that coding-agent evaluation percentages are only meaningful when a protocol file is written before the first task runs, warning that unpublished protocols allow datasets and scorers to shift, making results incomparable. The post proposes a YAML protocol template and a local scorer that enforces sealed splits and named metrics, emphasizing that changing a timeout after seeing a trace constitutes a new experiment requiring a new name.

A coding-agent percentage is a measurement only when a protocol file exists before the first task runs. Screenshots, live demos, and leaderboard rows can look precise while the dataset and the scorer still move. Reviewers should treat an unpublished protocol like a lab treats an unlabeled reagent on a crowded bench. The reading may be physically real, and it still cannot be compared with any later reading. The working analogy is a kitchen scale that is zeroed after the flour is already in the bowl. Moving the tare after the pour does not invent extra grain, yet it quietly rewrites every later claim. Agent evaluations repeat that gesture when tasks are added because a model looked weak on Tuesday. A frozen protocol does not make the agent smarter; it makes tomorrow's percentage talk about the same object. A useful dataset for this method is a holdout corpus with an explicit split, not a folder of favorite bugs. Each task needs a stable identifier, a language, a fixture path, a timeout, and a public-versus-hidden test flag. Tasks that authors used while prompting the agent belong in a development split and must never enter the reported score. The reported split should stay sealed until the protocol document is written, because curiosity itself is a contamination path. The protocol file is the original artifact of the method, and it should be boring on purpose. YAML is enough, because reviewers can diff it, hash it, and refuse results that lack the file. The example below is labeled as a template rather than as a published benchmark, since no live corpus is claimed here. protocol.yaml — template, not a published leaderboard schema: agent-eval-protocol/v1 name: sealed-holdout-demo created: "2026-09-09" splits: dev: "task-001", "task-002" report: "task-101", "task-102", "task-103" tasks: task-101: language: python fixture: fixtures/task-101.tar timeout s: 120 public tests: tests/public test.py hidden tests: tests/hidden test.py network: deny metrics: - hidden pass - wall clock ms - undeclared network - writes outside workspace aggregation: macro by language controls: dummy agent: empty diff comment agent: apply public hint toolchain lock: ".tool-versions" limits: max output bytes: 1048576 That file is the contract between the authors and anyone who will later cite the percentage. Changing a timeout after seeing a trace is not tuning; it is a new experiment that needs a new name. Teams that keep the YAML in git and refuse to score untagged commits already have most of this discipline. The remaining work is to make the scorer read the file instead of a hallway conversation. Metrics have to be named in that same document, or they will drift toward whichever number looks kind in a screenshot. Hidden-test pass rate is one metric, not a synonym for quality, and it should sit beside wall-clock time and undeclared network use. An agent that memorizes public tests the way a student memorizes an answer key will inflate pass rate while the companion metrics stay ugly. Aggregation is part of the metric: micro-average across tasks is not macro-average across languages, and mixing them after the run rewrites the claim. A small local scorer can enforce that contract without a platform and without a private dashboard. The Python below is a proposal for local use, not a report of production numbers. It should refuse to compute a percentage when the protocol hash is missing from the log. Labeled examples like this one are for method illustration and should not be cited as benchmark results. python harness score.py — proposal for a local sealed-split scorer import hashlib, json, subprocess, sys, time, yaml from collections import defaultdict from pathlib import Path def protocol hash path: Path - str: return hashlib.sha256 path.read bytes .hexdigest def load protocol path: Path - dict: doc = yaml.safe load path.read text encoding="utf-8" if "splits" not in doc or "report" not in doc "splits" : raise ValueError "protocol missing sealed report split" return doc def run agent cmd, workspace, timeout : start = time.monotonic proc = subprocess.run cmd, cwd=workspace, capture output=True, timeout=timeout elapsed ms = int time.monotonic - start 1000 return proc.returncode, elapsed ms def macro by language rows : bucket = defaultdict list for row in rows: bucket row "language" .append row "hidden pass" means = sum v / len v for v in bucket.values if v return sum means / len means if means else 0.0 if name == " main ": proto path = Path sys.argv 1 digest = protocol hash proto path protocol = load protocol proto path print json.dumps {"protocol sha256": digest, "aggregation": protocol "aggregation" } Controls are the unglamorous half of the method, and they are the half that marketing posts usually skip. The host should pin the toolchain and deny outbound network unless a task declares a need. It should always run a dummy agent that emits an empty diff before any candidate is scored. That dummy is a floor control, and a score above zero means the tests or the scorer are leaking. A second control can apply one obvious patch taken from a comment in the public tests. That weak agent estimates how much of the task was already solved in the prose of the tests. Together the floor and the weak ceiling give the later candidate percentage a scale, the way a photograph needs a ruler in the frame. Without those two control runs, a ninety percent row is a cropped photo, not a measurement. dummy agent.py — floor control: must score zero on a sealed hidden split from pathlib import Path import sys def main workspace: str - None: Write a marker only. A nonzero hidden pass means the harness is leaking. Path workspace, "AGENT RAN" .write text "dummy\n", encoding="utf-8" if name == " main ": main sys.argv 1 comment agent.py — weak ceiling: apply a hint only if public tests mention it import re, sys from pathlib import Path HINT = re.compile r"HINT:\s+ \S+ \s+- \s+ \S+ " def main workspace: str - None: root = Path workspace public = root.joinpath "tests", "public test.py" if not public.exists : return match = HINT.search public.read text encoding="utf-8" if not match: return src, dst = match.group 1 , match.group 2 src root = root.joinpath "src" if not src root.exists : return for path in src root.rglob " .py" : body = path.read text encoding="utf-8" if src in body: path.write text body.replace src, dst , encoding="utf-8" if name == " main ": main sys.argv 1 A short command sequence keeps the floor visible in the log before any candidate is allowed to run. Operators who skip the dummy run are asking the candidate agent to grade its own classroom. The commands below assume a Unix shell and a locked toolchain file sitting beside the protocol. Printing the hashes next to the dummy score makes later candidate rows harder to detach from the protocol. sha256sum protocol.yaml .tool-versions python dummy agent.py ./workspaces/task-101 python harness score.py protocol.yaml Continue only if dummy hidden pass is exactly 0.0 on the report split. python comment agent.py ./workspaces/task-101 python harness score.py protocol.yaml When a local machine cannot host the candidate agent, the protocol file should travel with the run. Rewriting the protocol for a new host creates a second experiment under the same percentage label. MonkeyCode currently offers free model access and a free server option for teams that need a spare runner. Disclosure: This article was prepared as part of MonkeyCode's product outreach. A remote runner still has to print the protocol hash, the dummy floor, and the candidate rows in one log. If the server image cannot pin the toolchain, the comparison is a different experiment and needs a different protocol name. Pasting a failing assertion back into the agent breaks the holdout seal in a single keystroke. The cheap runner is useful when it preserves isolation, not when it becomes a second editor. The method does not detect training-set leakage against the public internet, and it does not certify fairness across languages or difficulty bands. Teams that need a sales slide by Friday should not use it, because a sealed holdout often lowers the published percentage. Groups without authority to freeze the task list will produce theater no matter how pretty the YAML looks. This approach also does not belong in suites that fail under the dummy agent on a second run. A number that survives the dummy floor, the comment ceiling, and a hashed protocol is still not a verdict on intelligence. It is a comparable observation, which is a narrower and more honest claim than a marketing row. Readers who already keep protocol files can ignore any vendor runner and keep the YAML. Readers who need a spare machine for the same sealed split may try that free server on their own holdout.