A coding-agent percentage remains marketing until protocol, canaries, and a cost ledger freeze beside it. Teams still quote a lonely pass rate as if that number could travel without a suitcase of hidden choices. Prompt text, tool allowlists, sandbox images, and grader prompts often move the score more than the model does. The honest unit of publication is therefore a registered protocol rather than a percentage standing alone on a slide.
The protocol resembles a flight plan much more than a souvenir photograph of cruising altitude. A photograph of thirty thousand feet says nothing about fuel load, weather, or whether the altimeter was calibrated. Agent leaderboards behave the same way when they publish altitude without publishing the plan that produced it. Pre-registration does not make models stronger; it only makes later numbers comparable to the original claim.
A useful registry records the corpus identifier, the hidden holdout rule, and the grader isolation boundary before scored runs. It also records decoding settings, the tool allowlist, and the network policy that the sandbox will enforce during execution. Those fields look bureaucratic until a rerun drifts after someone quietly upgrades a linter inside the evaluation image. The registry exists to make that drift visible to readers, not to decorate a repository README with extra YAML.
Dataset construction starts with task families that a human can grade without watching the agent think out loud. Each family needs a public cousin for debugging and a hidden cousin reserved until the prompt hash is locked. Difficulty labels are operator judgments and should be treated as metadata, not as a second score competing with pass rate. Duplicate statements of the same bug should collapse to one holdout item so the percentage does not double-count a skill.
Holdout tasks must be sampled after the system prompt is frozen, not before that freeze is committed. Otherwise the prompt is tuned against the same families that later appear inside the headline metric. University exam design offers a close analogy: writing the study guide after seeing the final is not measurement. A protocol that cannot state when the holdout was sampled is not ready to emit a public percentage.
Negative-control canaries sit beside that holdout as a pulse check on the harness rather than on the model. A canary is a task the operator already knows should fail, or should pass, for reasons independent of model quality. If a known-dead canary starts passing, the grader is leaking, the sandbox is dirty, or string matching is too generous. If a known-easy canary starts failing, the environment moved, and the later pass rate is describing that movement.
Pass rate remains a useful column in the report, but it cannot serve as the entire published result. A registered run should also carry attempt count, wall-clock time, and an approximate token ledger for agent and grader. Variance across seeds belongs in the same row, because a small gap inside the noise band is not a ranking. Cost belongs there because two agents that pass the same hidden tests at very different spend are not interchangeable.
None of those extra fields require a glamorous dashboard; they require a schema that refuses incomplete score files. Marketing copy prefers a single integer because a single integer is easy to screenshot and easy to rank. A methodology that wants citation rather than advertising treats a missing canary result as a hard error. The refusal is closer to a test suite that never ran than to a footnote about optional diagnostics.
Grader isolation remains the control most often skipped when a small team assembles a weekend harness. The agent should not see hidden tests, and the grader should not see chain of thought unless the protocol says so. Shared context is how a model appears to understand a constraint that never appeared in the user-visible prompt. Isolation is not hostility toward agents; it is the same wall that separates student answers from the answer key.
Controls include a frozen temperature, a capped tool-output budget, and a network policy that defaults to deny. The sandbox image digest belongs in the lockfile because package churn inside an image is an unlisted independent variable. Clock source matters for wall-time: measuring on a busy laptop mixes agent latency with thermal throttling and browser updates. The token ledger should separate agent tokens from grader tokens so a verbose judge cannot masquerade as model spend.
The following example is a proposed workflow, not a claim about any executed leaderboard or vendor result. It stores a protocol document, hashes that document, runs canaries first, and only then writes a scored pass rate. Operators can adapt the same shape to their own corpus; the point is the refusal path, not a particular framework.
schema_version: 1
protocol_id: agent-eval-2026-09-10
prompt_ref: prompts/system.v3.txt
corpus_ref: datasets/public_tasks.sha256
holdout_rule: sample_after_prompt_freeze
holdout_seed: 20260910
canaries:
- id: dead_import_cycle
expect: fail
- id: hello_fs_write
expect: pass
controls:
temperature: 0.0
max_tool_output_bytes: 65536
network: deny
tools: [edit, shell, pytest]
sandbox_image: eval-runner@sha256:replace-me
grader:
isolation: hidden_tests_only
sees_chain_of_thought: false
metrics_required:
- pass_rate
- n_tasks
- n_seeds
- wall_clock_s
- token_in_agent
- token_out_agent
- token_in_grader
- token_out_grader
- canary_status
A tiny registrar can refuse to proceed when the document is incomplete or when the hash has not been committed. The commands below are ordinary and local; they do not imply any particular vendor quota, model name, or hardware. The helper reads YAML, checks required keys, and writes a lockfile that later score files must cite. Operators should commit that lockfile before any holdout run that they intend to quote.
from __future__ import annotations
import hashlib
import json
import sys
from pathlib import Path
import yaml
REQUIRED = (
"protocol_id",
"prompt_ref",
"corpus_ref",
"holdout_rule",
"canaries",
"controls",
"grader",
"metrics_required",
)
def load_protocol(path: Path) -> dict:
data = yaml.safe_load(path.read_text())
missing = [key for key in REQUIRED if key not in data]
if missing:
raise SystemExit(f"protocol missing fields: {missing}")
if data.get("holdout_rule") != "sample_after_prompt_freeze":
raise SystemExit("holdout must be sampled after the prompt freeze")
if not data["canaries"]:
raise SystemExit("refusing to register a protocol without canaries")
return data
def digest(path: Path) -> str:
return hashlib.sha256(path.read_bytes()).hexdigest()
def main() -> None:
protocol_path = Path(sys.argv[1])
protocol = load_protocol(protocol_path)
record = {
"protocol_id": protocol["protocol_id"],
"protocol_sha256": digest(protocol_path),
"prompt_sha256": digest(Path(protocol["prompt_ref"])),
"corpus_sha256": Path(protocol["corpus_ref"]).read_text().strip(),
}
out = Path("protocol_lock.json")
out.write_text(json.dumps(record, indent=2) + "\n")
print(f"registered {record['protocol_id']} -> {out}")
if __name__ == "__main__":
main()
python register_protocol.py protocol.yaml
test -f protocol_lock.json
sha256sum protocol.yaml prompts/system.v3.txt
python -c "import json; print(json.load(open('protocol_lock.json'))['protocol_sha256'])"
git add protocol.yaml protocol_lock.json prompts/system.v3.txt
git commit -m "lock evaluation protocol before scored runs"
The runner should treat canaries as a gate rather than as extra rows averaged into the headline percentage. Averaging a known-dead task into the pass rate hides harness failure inside what looks like model failure. The sketch below prints a score only after every canary matches its expected pass or fail polarity. Replace the stubbed sandbox call before any real use; the article does not report executed benchmark numbers.
from __future__ import annotations
import json
import time
from pathlib import Path
def run_task(task_id: str) -> dict:
"""Replace with a real sandbox call. This stub is a shape, not a result."""
raise NotImplementedError("wire this to an isolated runner")
def assert_canaries(canaries: list[dict]) -> None:
for spec in canaries:
result = run_task(spec["id"])
passed = bool(result["passed"])
should_pass = spec["expect"] == "pass"
if passed != should_pass:
raise SystemExit(
f"canary {spec['id']} broke the harness; refusing score"
)
def summarize(results: list[dict], started: float) -> dict:
n = len(results)
passes = sum(1 for row in results if row["passed"])
return {
"n_tasks": n,
"pass_rate": None if n == 0 else passes / n,
"wall_clock_s": time.time() - started,
"token_in_agent": sum(row.get("token_in_agent", 0) for row in results),
"token_out_agent": sum(row.get("token_out_agent", 0) for row in results),
"token_in_grader": sum(row.get("token_in_grader", 0) for row in results),
"token_out_grader": sum(row.get("token_out_grader", 0) for row in results),
"canary_status": "ok",
}
def main() -> None:
lock = json.loads(Path("protocol_lock.json").read_text())
protocol = json.loads(Path("protocol_expanded.json").read_text())
required = {"pass_rate", "n_tasks", "wall_clock_s", "canary_status"}
assert_canaries(protocol["canaries"])
started = time.time()
results = [run_task(task_id) for task_id in protocol["holdout_task_ids"]]
summary = summarize(results, started)
missing = sorted(required - set(summary))
if missing:
raise SystemExit(f"score file missing metrics: {missing}")
summary["protocol_sha256"] = lock["protocol_sha256"]
Path("score.json").write_text(json.dumps(summary, indent=2) + "\n")
if __name__ == "__main__":
main()
The important branch is the hard exit on a mismatched canary, which keeps a dirty sandbox from looking like progress. Without that branch, the rest of the file is only a pretty printer for whatever the environment happened to do. With it, an upgraded linter, a leaked test file, or a broken import cannot mint a number that resembles improvement. Readers who cannot reproduce the exit condition cannot reproduce the claim, regardless of how tidy the percentage looks.
Local laptops are poor clocks for wall-time comparisons and poor isolation boundaries when the agent receives a shell. A remote runner with a known image digest is closer to a laboratory bench than to a developer notebook full of extras. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option that can host this gated harness off a single workstation.
That pairing matters because protocol hashes are weak if every rerun happens on a different unmanaged laptop. The free server option is a stable place to execute the locked document, not a substitute for canaries or isolation. The free model access is only one interchangeable backend behind the same allowlist and the same hidden tests. Swapping backends without swapping the protocol is the entire practical point of registration in this workflow.
Marketing selects a peak result, then discards the envelope that made the peak possible to observe. A registered protocol publishes that envelope, including the canaries that would have forbidden the run entirely. Readers can disagree with the corpus and still reproduce the envelope, which differs from arguing with a screenshot. If a later post omits the protocol hash, the cost ledger, or the canary gate, the percentage has returned to advertising.
The method remains narrow and should not be sold as a general verdict on coding agents or developer productivity. It does not estimate real-user utility, and it does not certify safety properties of tools that execute arbitrary code. It also does not rescue a corpus that is too small or too close to widely circulated training material. Teams that need a launch headline, or that cannot isolate a grader from the agent trace, should not use this workflow.
Researchers comparing decoding settings and platform engineers checking sandbox drift are the intended readers of this protocol. They already suspect that a lonely pass rate is a costume worn by an unpublished bundle of choices. Pre-registration simply gives that suspicion a file format, a hash, and a refusal path when a dead canary stands up. The percentage can come last, after the lockfile exists, which is the opposite order from a marketing screenshot.