My LLM Critic Flip-Flops on Every Run. That's Fine — Because a Frozenset Decides What's Fatal.

A developer's LLM-based plan critic returned a different verdict on every trial of identical input, yet never let a defective plan through, because deterministic gates and a code-enforced allowlist, not the LLM, own the safety contract. The frozenset of blocker-eligible families downgrades any LLM severity misclassification before it enters the findings list, achieving zero under-claim approvals despite a 100% label flip rate.

This is a companion to the PlannerCritic series . Article 2 was about a specific critic bug. This one is about the design principle I extracted from fixing it — and the measurement that proved it holds.I measured my LLM critic on identical input across five trials. It returned a different verdict every single time. label flip rate = 1.0. It also never let a defective plan through. underclaim approvals = 0. Both are true. The frozenset is why. In v0.2.1 I added a test I'd been avoiding: send the same boundary-case plans through the real critic model five times and measure what changes. The live-critic boundary evaluator https://github.com/deghosal-2026/planner-critic-engine/blob/main/src/planner critic/eval/live boundary.py 218 . The numbers: | Metric | Value | What it means | |---|---|---| label flip rate | 1.000 | The critic changes its verdict on every trial of identical input | evidence drift rate | 1.000 | The critic invents a different explanation every trial | family migration rate | 0.000 | No seeded defect landed in an advisory family | underclaim approvals | 0 | No defective plan got zero blockers | Read those first two rows carefully. On the same plan , five times in a row, the critic returned a different verdict and a different reasoning every time. If you were betting the safety contract on the critic being consistent, you'd have lost. I wasn't betting on that. But it was still uncomfortable to see it measured at 1.0. "Mostly consistent" would have felt safer than "maximally inconsistent." Maximally inconsistent is what we got. The critic is 100% non-deterministic on verdict . It is 0% under-claiming on seeded defects . Both numbers are real. The reason both can be true at once is that they measure different directions, and the architecture assigns them to different owners. There are two ways an LLM critic can be wrong: The architecture gives each direction to a different authority, and neither authority is the LLM. The deterministic gates own the under-claim direction. Preconditions, topological ordering, rollback credibility, verification ordering — these parse the plan's AST, not its prose. They cannot be prompt-injected because they don't read natural language Article 5 https://dev.to/debashish ghosal/i-tried-to-prompt-inject-my-own-agent-engine-it-didnt-work-heres-why-57m0 covers this . A defective plan that the LLM critic happens to miss on trial 3 still gets caught by the gate that checks whether every precondition is established by an earlier task. That's why underclaim approvals = 0 despite label flip rate = 1.0 . A code-enforced allowlist owns the over-claim direction. This is the part I want to dwell on, because it's the part I learned the hard way. In Article 2 https://dev.to/debashish ghosal/i-told-my-llm-critic-to-be-adversarial-it-started-blocking-plans-for-being-not-thorough-enough-172 I told the critic to be "an adversarial plan reviewer." It obeyed. It blocked plans for being incomplete — "this plan could also cover edge case X" — not for being unsafe. Every strict goal escalated for the wrong reason. The fix wasn't more prompt engineering. I tried that first; the critic still escalated completeness concerns to blocker about 30% of the time. The fix was a frozenset: BLOCKER ELIGIBLE FAMILIES = frozenset { "unsafe sequencing", "weak rollback", "unverified dependencies", "feasibility", } if severity == Severity.BLOCKER and item.heuristic family not in BLOCKER ELIGIBLE FAMILIES: severity = Severity.WARNING Even if the LLM returns blocker for a risk or missing steps finding, the code downgrades it before it enters the findings list. After the fix, zero advisory findings appeared as blockers across 92 post-fix runs. The prompt is helpful. The frozenset is the contract. That's the sentence I kept coming back to. The LLM is allowed to be wrong about severity — and it is, on every trial — because the code doesn't trust the LLM's severity label. It trusts the structural property which family the finding is in , which the gate derived deterministically. The LLM's label is decorative; the family is load-bearing. Here's the part I think generalizes beyond my project. When you put LLM judgment on a critical path, you inherit every vulnerability of LLM judgment — non-determinism, prompt sensitivity, the tendency to be "thorough" when you asked it to be "adversarial." When you keep the critical path deterministic, you get resistance by design — but only to the things your code can check structurally. The split that worked: The mistake is putting the LLM in charge of the second category and making its verdict load-bearing. The fix is letting the LLM inform the second category while code decides whether its verdict counts. The frozenset is the mechanism: the LLM proposes a severity; code checks whether the structural property supports it; code wins ties. You can see the same pattern in how approval works. In approval.py https://github.com/deghosal-2026/planner-critic-engine/blob/main/src/planner critic/approval.py , a gate blocker is always a hard blocker; an LLM critic blocker is probabilistic and posture-dependent. The gate vetoes are deterministic. The critic's contribution is downgraded to advisory under postures that can't afford false escalations. I want to be careful not to oversell this. The deterministic authority has a real limit, and it's the one a commenter @ethanwritesai sharpened for me: the frozenset only works because the family a finding belongs to is itself derived deterministically. If the LLM could mislabel which family its finding is in, the allowlist would be trusting a label again. Right now the family comes from the gate or the critic's structured output, and the critic's structured output is the soft edge. That's tracked now — the property-vs-label distinction is part of the v0.3.0 critic-satisfaction work 254 https://github.com/deghosal-2026/planner-critic-engine/issues/254 . The principle holds; the implementation has a seam where the LLM's self-classification still leaks in. I'd rather name it than hide it. There's a second, more obvious seam: the deterministic gates only check structure , so a well-formed malicious plan — dummy rollback, dummy verification — satisfies the linter. Deterministic authority is necessary; it is not sufficient. That seam and the indirect-injection surface that feeds it is the subject of its own piece — see I Published Every Flaw My Safety Tool Can't Catch https://dev.to/debashish ghosal/i-published-every-flaw-my-safety-tool-cant-catch . I'm keeping this article about the frozenset and that one about the holes. I stopped trying to make the critic consistent. I spent a while in v0.1.0 tuning the prompt to get stable verdicts. It didn't work, and the boundary evaluator finally told me why in numbers: the ceiling is 1.0, not because the prompt is bad but because the model is non-deterministic by construction. You don't fix that with prompt engineering. You route around it by not depending on it. I also stopped treating "the critic flagged something" as a reason to escalate. Under balanced posture, a critic blocker is a warning, not a veto — because the critic is allowed to be wrong in the over-claim direction, and the cost of a false escalation is a human's time. The deterministic gates are the veto layer. The critic is the "you should probably look at this" layer. And I stopped trusting the LLM to decide what was fatal. That's the whole title of this piece. The LLM is allowed an opinion about fatality. The frozenset decides. I don't think deterministic-first is the whole answer. I think it's the floor. The interesting work is what you build on top of the floor — and how honest you are about where the floor ends. Series: Article 1 https://dev.to/debashish ghosal/i-ran-157-agent-plans-against-a-real-llm-the-problem-wasnt-execution-it-was-planning-163j · Article 2 https://dev.to/debashish ghosal/i-told-my-llm-critic-to-be-adversarial-it-started-blocking-plans-for-being-not-thorough-enough-172 · Article 3 https://dev.to/debashish ghosal/the-planner-made-the-same-3-mistakes-every-time-a-bigger-model-didnt-fix-it-3170 · Article 4 https://dev.to/debashish ghosal/i-ran-157-agent-goals-for-030-the-field-test-found-10-issues-that-unit-tests-never-would-hgk · Article 5 https://dev.to/debashish ghosal/i-tried-to-prompt-inject-my-own-agent-engine-it-didnt-work-heres-why-57m0 Links: critique/critic.py approval.py eval/live boundary.py