{"slug": "the-gate-that-stayed-silent-when-a-blocker-count-that-drops-reads-as-improvement", "title": "The Gate That Stayed Silent — When a Blocker Count That Drops Reads as Improvement", "summary": "A developer identified a critical failure mode in their deterministic safety gates for AI plan evaluation: when a gate stops firing, blocker counts drop and the improvement masks a regression. The fix is a gate canary, a deterministic health check that runs before every eval sweep to assert each gate still fires on known-bad plans, ensuring silent gate death is detected.", "body_md": "In the\n\n[last article], I argued that the safety contract should move out of the LLM critic and into deterministic gates. The critic can vary. The gates should hold.\n\nThis one is about the failure mode I did not see coming: what happens when a gate stops working and nobody notices because the metrics look better.\n\nBy `v0.2.2`\n\n, the blocker counts were dominated by structural families:\n\n`unsafe_sequencing`\n\n: `unverified_dependencies`\n\n: `weak_rollback`\n\n: Those numbers felt like evidence that the deterministic layer was doing its job. A high blocker count on structural categories meant the gates were firing.\n\nThen a community reader named **Artjoms Stukans** pointed out the failure mode I had not modeled:\n\n\"If one blocker class stops firing after some refactor, your numbers only look better. 226 becomes 40 and that reads like plans got safer.\"\n\nHe described a Kubernetes incident where four releases in a row never actually ran. An old ReplicaSet kept one pod Running. Every health check passed. Every smoke test was green. Nothing said a word.\n\nThat is exactly the shape of this problem.\n\nA deterministic gate is more reliable than an LLM critic. That is true. But it is also quieter. A non-deterministic critic advertises its own unreliability through label flips and evidence drift. A gate that silently stops firing fails the same way on every trial. Repeated runs agree. The agreement reads as confidence. You lose the variance signal exactly where the contract now lives.\n\nBefore Artjoms's comment, I would have said the system already had protection against this.\n\nIt did. Just not enough.\n\n| Mechanism | What it detects | Why it misses silent gate death |\n|---|---|---|\n`underclaim_approvals` metric |\nLLM critic failed to block a known-bad plan | Tests the LLM critic, not the deterministic gates\n|\n| Deterministic gate unit tests | Each gate fires on its fixture | CI tests pass locally; the concern is production refactor |\n| Regression detector | New blockers appearing between revisions | Detects blockers appearing, not blockers disappearing\n|\n| Drift alert | Z-score spike in severity | Tracks severity changes, not total disappearance |\n\nThe gap was architectural: **no mechanism asserted that each gate class still fires on a known-bad plan as a continuous health check**. The gates were tested at build time. They were not monitored at runtime.\n\nThe fix is a gate canary — a deterministic, zero-LLM-cost health check that runs before every eval sweep and asserts each gate still fires on its canary bad plan.\n\nEach gate class gets a `(good_plan, bad_plan)`\n\npair:\n\n| Gate | Good plan | Bad plan |\n|---|---|---|\n`ordering` |\nt1 → t2 | t2 → t1 |\n`rollback_credible` |\nhigh risk + reachable rollback | high risk + unreachable rollback |\n`verification_ordering` |\nverify before consume | consume before verified |\n`preconditions` |\nprecondition references established fact | precondition references nothing |\n\nThe canary runs as:\n\n`plancritic gates canary --check`\n\n— fails CI if any gate stopped firingIf a refactor or a bug silences the `ordering`\n\ngate, the canary turns red immediately. The blocker count dropping from 226 to 40 no longer reads as improvement. It reads as a regression that must be explained.\n\nThe gate canary fixes the immediate gap. The harder lesson is about trust models.\n\nI had been thinking about safety in two layers:\n\nThat framing is not wrong, but it is incomplete. The gates are stable only if someone watches them. Trusting a deterministic layer because it is deterministic is the same mistake as trusting a test suite because the tests pass. The question is not whether the gates are reliable by construction. The question is whether you know the moment they stop being reliable.\n\n**A deterministic safety layer does not remove the monitoring problem. It moves it.**\n\nBefore the fix, the monitoring was on the critic's non-determinism (label_flip_rate, evidence_drift). After the fix, the monitoring has to be on the gates themselves — not on their outputs, but on their continued ability to fire.\n\nThat is what the gate canary does. It does not check whether the gate approved or blocked a particular plan. It checks whether the gate is still alive.\n\nIf you build deterministic safety layers for LLM systems, ask yourself:\n\n**What is your equivalent of the Kubernetes ReplicaSet that kept one pod Running?**\n\nWhat component, if it stopped working, would produce better-looking metrics?\n\nThe answer is rarely obvious at design time. I only found mine because a reader described an incident from an unrelated system and the shape matched.\n\nThe general principle is:\n\nThat is the difference between a safety layer you trust and a safety layer you have not yet watched break.\n\n**Previous PlannerCritic articles**\n\n**Links**\n\n**Next in the sequence:** *Two Projects, One Problem — What PlannerCritic and AdversarialDebate Each Got Wrong*", "url": "https://wpnews.pro/news/the-gate-that-stayed-silent-when-a-blocker-count-that-drops-reads-as-improvement", "canonical_source": "https://dev.to/debashish_ghosal/the-gate-that-stayed-silent-when-a-blocker-count-that-drops-reads-as-improvement-3je9", "published_at": "2026-08-31 04:20:50+00:00", "updated_at": "2026-08-31 04:51:37.795829+00:00", "lang": "en", "topics": ["ai-safety", "developer-tools"], "entities": ["Artjoms Stukans"], "alternates": {"html": "https://wpnews.pro/news/the-gate-that-stayed-silent-when-a-blocker-count-that-drops-reads-as-improvement", "markdown": "https://wpnews.pro/news/the-gate-that-stayed-silent-when-a-blocker-count-that-drops-reads-as-improvement.md", "text": "https://wpnews.pro/news/the-gate-that-stayed-silent-when-a-blocker-count-that-drops-reads-as-improvement.txt", "jsonld": "https://wpnews.pro/news/the-gate-that-stayed-silent-when-a-blocker-count-that-drops-reads-as-improvement.jsonld"}}