{"slug": "canary-gates-make-agent-rankings-comparable", "title": "Canary Gates Make Agent Rankings Comparable", "summary": "A developer proposes a registered-protocol approach to publishing coding-agent benchmark scores, arguing that a bare pass rate is meaningless without frozen prompts, hidden holdouts, negative-control canaries, and a cost ledger. The protocol requires recording corpus identifiers, holdout sampling rules, grader isolation boundaries, decoding settings, tool allowlists, and sandbox network policy before scored runs, and treats a missing canary result as a hard error rather than an optional diagnostic.", "body_md": "A coding-agent percentage remains marketing until protocol, canaries, and a cost ledger freeze beside it. Teams still quote a lonely pass rate as if that number could travel without a suitcase of hidden choices. Prompt text, tool allowlists, sandbox images, and grader prompts often move the score more than the model does. The honest unit of publication is therefore a registered protocol rather than a percentage standing alone on a slide.\n\nThe protocol resembles a flight plan much more than a souvenir photograph of cruising altitude. A photograph of thirty thousand feet says nothing about fuel load, weather, or whether the altimeter was calibrated. Agent leaderboards behave the same way when they publish altitude without publishing the plan that produced it. Pre-registration does not make models stronger; it only makes later numbers comparable to the original claim.\n\nA useful registry records the corpus identifier, the hidden holdout rule, and the grader isolation boundary before scored runs. It also records decoding settings, the tool allowlist, and the network policy that the sandbox will enforce during execution. Those fields look bureaucratic until a rerun drifts after someone quietly upgrades a linter inside the evaluation image. The registry exists to make that drift visible to readers, not to decorate a repository README with extra YAML.\n\nDataset construction starts with task families that a human can grade without watching the agent think out loud. Each family needs a public cousin for debugging and a hidden cousin reserved until the prompt hash is locked. Difficulty labels are operator judgments and should be treated as metadata, not as a second score competing with pass rate. Duplicate statements of the same bug should collapse to one holdout item so the percentage does not double-count a skill.\n\nHoldout tasks must be sampled after the system prompt is frozen, not before that freeze is committed. Otherwise the prompt is tuned against the same families that later appear inside the headline metric. University exam design offers a close analogy: writing the study guide after seeing the final is not measurement. A protocol that cannot state when the holdout was sampled is not ready to emit a public percentage.\n\nNegative-control canaries sit beside that holdout as a pulse check on the harness rather than on the model. A canary is a task the operator already knows should fail, or should pass, for reasons independent of model quality. If a known-dead canary starts passing, the grader is leaking, the sandbox is dirty, or string matching is too generous. If a known-easy canary starts failing, the environment moved, and the later pass rate is describing that movement.\n\nPass rate remains a useful column in the report, but it cannot serve as the entire published result. A registered run should also carry attempt count, wall-clock time, and an approximate token ledger for agent and grader. Variance across seeds belongs in the same row, because a small gap inside the noise band is not a ranking. Cost belongs there because two agents that pass the same hidden tests at very different spend are not interchangeable.\n\nNone of those extra fields require a glamorous dashboard; they require a schema that refuses incomplete score files. Marketing copy prefers a single integer because a single integer is easy to screenshot and easy to rank. A methodology that wants citation rather than advertising treats a missing canary result as a hard error. The refusal is closer to a test suite that never ran than to a footnote about optional diagnostics.\n\nGrader isolation remains the control most often skipped when a small team assembles a weekend harness. The agent should not see hidden tests, and the grader should not see chain of thought unless the protocol says so. Shared context is how a model appears to understand a constraint that never appeared in the user-visible prompt. Isolation is not hostility toward agents; it is the same wall that separates student answers from the answer key.\n\nControls include a frozen temperature, a capped tool-output budget, and a network policy that defaults to deny. The sandbox image digest belongs in the lockfile because package churn inside an image is an unlisted independent variable. Clock source matters for wall-time: measuring on a busy laptop mixes agent latency with thermal throttling and browser updates. The token ledger should separate agent tokens from grader tokens so a verbose judge cannot masquerade as model spend.\n\nThe following example is a proposed workflow, not a claim about any executed leaderboard or vendor result. It stores a protocol document, hashes that document, runs canaries first, and only then writes a scored pass rate. Operators can adapt the same shape to their own corpus; the point is the refusal path, not a particular framework.\n\n```\n# protocol.yaml — proposed schema, not a published result\nschema_version: 1\nprotocol_id: agent-eval-2026-09-10\nprompt_ref: prompts/system.v3.txt\ncorpus_ref: datasets/public_tasks.sha256\nholdout_rule: sample_after_prompt_freeze\nholdout_seed: 20260910\ncanaries:\n  - id: dead_import_cycle\n    expect: fail\n  - id: hello_fs_write\n    expect: pass\ncontrols:\n  temperature: 0.0\n  max_tool_output_bytes: 65536\n  network: deny\n  tools: [edit, shell, pytest]\n  sandbox_image: eval-runner@sha256:replace-me\ngrader:\n  isolation: hidden_tests_only\n  sees_chain_of_thought: false\nmetrics_required:\n  - pass_rate\n  - n_tasks\n  - n_seeds\n  - wall_clock_s\n  - token_in_agent\n  - token_out_agent\n  - token_in_grader\n  - token_out_grader\n  - canary_status\n```\n\nA tiny registrar can refuse to proceed when the document is incomplete or when the hash has not been committed. The commands below are ordinary and local; they do not imply any particular vendor quota, model name, or hardware. The helper reads YAML, checks required keys, and writes a lockfile that later score files must cite. Operators should commit that lockfile before any holdout run that they intend to quote.\n\n```\n# register_protocol.py — example helper, unexecuted in this article\nfrom __future__ import annotations\n\nimport hashlib\nimport json\nimport sys\nfrom pathlib import Path\n\nimport yaml\n\nREQUIRED = (\n    \"protocol_id\",\n    \"prompt_ref\",\n    \"corpus_ref\",\n    \"holdout_rule\",\n    \"canaries\",\n    \"controls\",\n    \"grader\",\n    \"metrics_required\",\n)\n\ndef load_protocol(path: Path) -> dict:\n    data = yaml.safe_load(path.read_text())\n    missing = [key for key in REQUIRED if key not in data]\n    if missing:\n        raise SystemExit(f\"protocol missing fields: {missing}\")\n    if data.get(\"holdout_rule\") != \"sample_after_prompt_freeze\":\n        raise SystemExit(\"holdout must be sampled after the prompt freeze\")\n    if not data[\"canaries\"]:\n        raise SystemExit(\"refusing to register a protocol without canaries\")\n    return data\n\ndef digest(path: Path) -> str:\n    return hashlib.sha256(path.read_bytes()).hexdigest()\n\ndef main() -> None:\n    protocol_path = Path(sys.argv[1])\n    protocol = load_protocol(protocol_path)\n    record = {\n        \"protocol_id\": protocol[\"protocol_id\"],\n        \"protocol_sha256\": digest(protocol_path),\n        \"prompt_sha256\": digest(Path(protocol[\"prompt_ref\"])),\n        \"corpus_sha256\": Path(protocol[\"corpus_ref\"]).read_text().strip(),\n    }\n    out = Path(\"protocol_lock.json\")\n    out.write_text(json.dumps(record, indent=2) + \"\\n\")\n    print(f\"registered {record['protocol_id']} -> {out}\")\n\nif __name__ == \"__main__\":\n    main()\npython register_protocol.py protocol.yaml\ntest -f protocol_lock.json\nsha256sum protocol.yaml prompts/system.v3.txt\npython -c \"import json; print(json.load(open('protocol_lock.json'))['protocol_sha256'])\"\ngit add protocol.yaml protocol_lock.json prompts/system.v3.txt\ngit commit -m \"lock evaluation protocol before scored runs\"\n```\n\nThe runner should treat canaries as a gate rather than as extra rows averaged into the headline percentage. Averaging a known-dead task into the pass rate hides harness failure inside what looks like model failure. The sketch below prints a score only after every canary matches its expected pass or fail polarity. Replace the stubbed sandbox call before any real use; the article does not report executed benchmark numbers.\n\n```\n# run_gated_eval.py — proposed control flow, not an executed result\nfrom __future__ import annotations\n\nimport json\nimport time\nfrom pathlib import Path\n\ndef run_task(task_id: str) -> dict:\n    \"\"\"Replace with a real sandbox call. This stub is a shape, not a result.\"\"\"\n    raise NotImplementedError(\"wire this to an isolated runner\")\n\ndef assert_canaries(canaries: list[dict]) -> None:\n    for spec in canaries:\n        result = run_task(spec[\"id\"])\n        passed = bool(result[\"passed\"])\n        should_pass = spec[\"expect\"] == \"pass\"\n        if passed != should_pass:\n            raise SystemExit(\n                f\"canary {spec['id']} broke the harness; refusing score\"\n            )\n\ndef summarize(results: list[dict], started: float) -> dict:\n    n = len(results)\n    passes = sum(1 for row in results if row[\"passed\"])\n    return {\n        \"n_tasks\": n,\n        \"pass_rate\": None if n == 0 else passes / n,\n        \"wall_clock_s\": time.time() - started,\n        \"token_in_agent\": sum(row.get(\"token_in_agent\", 0) for row in results),\n        \"token_out_agent\": sum(row.get(\"token_out_agent\", 0) for row in results),\n        \"token_in_grader\": sum(row.get(\"token_in_grader\", 0) for row in results),\n        \"token_out_grader\": sum(row.get(\"token_out_grader\", 0) for row in results),\n        \"canary_status\": \"ok\",\n    }\n\ndef main() -> None:\n    lock = json.loads(Path(\"protocol_lock.json\").read_text())\n    protocol = json.loads(Path(\"protocol_expanded.json\").read_text())\n    required = {\"pass_rate\", \"n_tasks\", \"wall_clock_s\", \"canary_status\"}\n    assert_canaries(protocol[\"canaries\"])\n    started = time.time()\n    results = [run_task(task_id) for task_id in protocol[\"holdout_task_ids\"]]\n    summary = summarize(results, started)\n    missing = sorted(required - set(summary))\n    if missing:\n        raise SystemExit(f\"score file missing metrics: {missing}\")\n    summary[\"protocol_sha256\"] = lock[\"protocol_sha256\"]\n    Path(\"score.json\").write_text(json.dumps(summary, indent=2) + \"\\n\")\n\nif __name__ == \"__main__\":\n    main()\n```\n\nThe important branch is the hard exit on a mismatched canary, which keeps a dirty sandbox from looking like progress. Without that branch, the rest of the file is only a pretty printer for whatever the environment happened to do. With it, an upgraded linter, a leaked test file, or a broken import cannot mint a number that resembles improvement. Readers who cannot reproduce the exit condition cannot reproduce the claim, regardless of how tidy the percentage looks.\n\nLocal laptops are poor clocks for wall-time comparisons and poor isolation boundaries when the agent receives a shell. A remote runner with a known image digest is closer to a laboratory bench than to a developer notebook full of extras. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option that can host this gated harness off a single workstation.\n\nThat pairing matters because protocol hashes are weak if every rerun happens on a different unmanaged laptop. The free server option is a stable place to execute the locked document, not a substitute for canaries or isolation. The free model access is only one interchangeable backend behind the same allowlist and the same hidden tests. Swapping backends without swapping the protocol is the entire practical point of registration in this workflow.\n\nMarketing selects a peak result, then discards the envelope that made the peak possible to observe. A registered protocol publishes that envelope, including the canaries that would have forbidden the run entirely. Readers can disagree with the corpus and still reproduce the envelope, which differs from arguing with a screenshot. If a later post omits the protocol hash, the cost ledger, or the canary gate, the percentage has returned to advertising.\n\nThe method remains narrow and should not be sold as a general verdict on coding agents or developer productivity. It does not estimate real-user utility, and it does not certify safety properties of tools that execute arbitrary code. It also does not rescue a corpus that is too small or too close to widely circulated training material. Teams that need a launch headline, or that cannot isolate a grader from the agent trace, should not use this workflow.\n\nResearchers comparing decoding settings and platform engineers checking sandbox drift are the intended readers of this protocol. They already suspect that a lonely pass rate is a costume worn by an unpublished bundle of choices. Pre-registration simply gives that suspicion a file format, a hash, and a refusal path when a dead canary stands up. The percentage can come last, after the lockfile exists, which is the opposite order from a marketing screenshot.", "url": "https://wpnews.pro/news/canary-gates-make-agent-rankings-comparable", "canonical_source": "https://dev.to/apppro_5726/canary-gates-make-agent-rankings-comparable-knb", "published_at": "2026-09-10 04:34:57+00:00", "updated_at": "2026-09-10 04:49:34.986341+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "ai-safety", "developer-tools", "mlops"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/canary-gates-make-agent-rankings-comparable", "markdown": "https://wpnews.pro/news/canary-gates-make-agent-rankings-comparable.md", "text": "https://wpnews.pro/news/canary-gates-make-agent-rankings-comparable.txt", "jsonld": "https://wpnews.pro/news/canary-gates-make-agent-rankings-comparable.jsonld"}}