{"slug": "do-guardrails-make-ai-coding-agents-better-what-270-runs-showed", "title": "Do Guardrails Make AI Coding Agents Better? What 270 Runs Showed", "summary": "An experiment of 270 unattended coding-agent runs across three models and seven tasks found that guardrails did not raise first-pass pass rates: all 270 runs were accepted and 267 passed the hidden acceptance tests at first \"done\" (90 of 90 with no harness, 89 of 90 with an instruction file, 88 of 90 with gates). The harness showed up instead in defects left at \"done\" on the four harder tasks, where 9 of 60 runs with gates left a defect against 15 of 60 with none — a gap the author called too small in this sample to call, with the widest model gap (Sonnet, 1 versus 5) giving p = 0.18. The runs used Claude Code 2.1.296 on 13 and 14 October 2026 and cost an estimated $115.58 at list prices, while cost per accepted change was driven more than tenfold by model choice ($0.06 for Haiku versus $0.97 for Opus with no harness) than by the guardrails.", "body_md": "Every team that works with a coding agent ends up having the same argument. Someone has built the scaffolding: an instruction file, a pre-commit gate, a hook that refuses “done” until the tests have run. Someone else notices that the agent passes the tests anyway.\n\nThen the gate refuses the agent six times in one morning, and someone asks out loud: do the guardrails pay, or are they ceremony? It usually gets settled by conviction, because the costs arrive as waiting and the benefits as problems that never happened.\n\nSo for the book I ran an experiment.\n\nIn 270 unattended runs of a coding agent, across three models and seven tasks, every change was eventually accepted, and the guardrails did not raise first-pass rates. They showed up in what runs left behind at “done”: on the four later tasks, 9 of 60 runs with gates left a defect against 15 of 60 with none, a gap too small in this sample to call.\n\n## One run that passed every test\n\nThe task, T7, asked for recurring invoices. One run, with no instruction file and no hooks, passed the hidden acceptance tests at the first attempt. Then the project’s own checks ran on what it left behind, and one failed: it had changed the database schema without the test the team’s rule requires for every schema change, a rule this run never saw.\n\n## How the experiment was set up\n\nI ran them on 13 and 14 October 2026 with Claude Code 2.1.296, five runs per task, condition and model, interleaved: Sonnet and Opus on three everyday features (T1 to T3), then Haiku, Sonnet and Opus on four harder tasks (T4 to T7). As the diagram below shows, **none** got the task and the code, **rules** added an instruction file, and **gates** added a pre-commit gate and a hook that refuses “done” without fresh test evidence. The acceptance tests were hidden; on T4 to T7, any failure of the project’s own checks counted as a **defect at done**.\n\n## Did guardrails raise the pass rate?\n\nNo. All 270 runs were accepted, and 267 passed the hidden tests at the first “done”: none 90 of 90, rules 89 of 90, gates 88 of 90. The statistical tests I wrote down in advance found no clear difference between conditions for any model.\n\n## Where the harness did show up\n\nBack to T7. Without the instruction file, 14 of 15 T7 runs changed the schema without its test, whichever model ran; with the rule, none of 30 did. The rule says a migration and its database test change together: the Also-Update List (P04), Chapter 3, measured ([more on such files](https://secondtimeisabug.com/blog/agents-md-best-practices/)).\n\nPooled over T4 to T7, defects at done were 15 of 60 with none, 14 of 60 with rules and 9 of 60 with gates. Sonnet showed the widest gap, 1 against 5, and even that gave p = 0.18: a lead, not proof.\n\nOne caveat: T4 cannot be done without changing an existing test, and the instruction file says tests are read-only for agents. Under that rule, 23 of 30 runs built the feature, left the old test failing and asked a person to change it, which accounts for every defect under rules and gates. The rule was right ([why it exists](https://secondtimeisabug.com/blog/stop-ai-agent-deleting-tests/) is Chapter 6, a free [sample chapter](https://secondtimeisabug.com/sample/the-second-time-is-a-bug-chapter-6.pdf)), but nobody came, and “asked a person” counted the same as “left a bug”.\n\n## What did each accepted change cost?\n\nThe Cost per Accepted Change (P32), Chapter 12, divides the cost of every run, failed ones included, by the changes accepted. On T4 to T7, 20 runs per cell:\n\n| Model | none | rules | gates | \n|---|---|---|---|\n| Haiku | $0.06 | $0.04 | $0.05 | \n| Sonnet | $0.27 | $0.28 | $0.30 | \n| Opus | $0.97 | $0.84 | $0.82 | \n\nThe harness barely moved the cost; the model moved it more than tenfold. All 270 runs together cost an estimated $115.58, the agent tool’s own estimate at list prices, not a bill.\n\n## Which guards are still paying their way?\n\nEvery guard is a bet, placed [the second time a mistake happens](https://secondtimeisabug.com/blog/ai-coding-agent-same-mistake/), that the model will make it again. Guards Pay Rent (P33), Chapter 12, prices each one: wait, noise and upkeep against real catches.\n\n## What these runs cannot tell you\n\nFive runs per cell is the protocol’s minimum, on seven tasks, one small codebase, one agent and three models from one vendor, as they were in October 2026. I wrote the tasks, tests and harness, which could favour the harness, and the defect measure uses its own guards.\n\n## Check it yourself\n\nEverything is in the [companion repository (opens in a new tab)](https://github.com/second-time-is-a-bug-book/book-companion): the run records in [`experiment/results/` (opens in a new tab)](https://github.com/second-time-is-a-bug-book/book-companion/tree/v1.0.0/experiment/results), the plan for T4 to T7 in [`BATCH-2.md` (opens in a new tab)](https://github.com/second-time-is-a-bug-book/book-companion/blob/v1.0.0/experiment/tasks/BATCH-2.md), written before any of those runs, and [`reproduce.ts` (opens in a new tab)](https://github.com/second-time-is-a-bug-book/book-companion/blob/v1.0.0/experiment/results/reproduce.ts), which recomputes every number the book quotes:\n\n```\nnode experiment/results/reproduce.ts\n```\n\nAnd the T7 run? Count passes and it was perfect; count what it left behind and it was one of fourteen that handed on a schema change with no test. Which answer you get depends on what you measure, and so does the argument about ceremony.\n\n## Questions people ask\n\n### Was the harness worth it?\n\nNot in pass rate or cost: first passes were 88 to 90 of 90 in every condition, and cost per accepted change sat within 15 cents for each model. It showed up in what runs left behind, at the price of runs that stopped for an absent person; whether that pays is a judgement call.\n\n### Which model gave the cheapest accepted change?\n\nHaiku: $0.04 to $0.06 per accepted change on T4 to T7, against $0.27 to $0.30 for Sonnet and $0.82 to $0.97 for Opus, with all 60 runs passing at the first “done”. It was enough for small, fully specified tasks, which says nothing about larger work.\n\n### Can I run the same experiment on my own code?\n\nYes: the repository holds the protocol, seven tasks with hidden acceptance tests and a scorer that counts failed runs in the cost per accepted change. Swap in tasks from your own backlog, run each condition at least five times, interleaved, and report the spread as well as the mean.", "url": "https://wpnews.pro/news/do-guardrails-make-ai-coding-agents-better-what-270-runs-showed", "canonical_source": "https://secondtimeisabug.com/blog/coding-agent-guardrails-270-runs/", "published_at": "2026-10-28 01:46:12+00:00", "updated_at": "2026-10-28 02:41:44.534941+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-safety"], "entities": ["Claude Code", "Anthropic", "Sonnet", "Opus", "Haiku"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/do-guardrails-make-ai-coding-agents-better-what-270-runs-showed", "markdown": "https://wpnews.pro/news/do-guardrails-make-ai-coding-agents-better-what-270-runs-showed.md", "text": "https://wpnews.pro/news/do-guardrails-make-ai-coding-agents-better-what-270-runs-showed.txt", "jsonld": "https://wpnews.pro/news/do-guardrails-make-ai-coding-agents-better-what-270-runs-showed.jsonld"}}