My LLM Critic Flip-Flops on Every Run. That's Fine — Because a Frozenset Decides What's Fatal.

This is a companion to the

[PlannerCritic series].[Article 2]was about a specific critic bug. This one is about the design principle I extracted from fixing it — and the measurement that proved it holds.I measured my LLM critic on identical input across five trials. It returned a different verdict every single time. label_flip_rate = 1.0. It also never let a defective plan through. underclaim_approvals = 0. Both are true. The frozenset is why.

In v0.2.1 I added a test I'd been avoiding: send the same boundary-case plans through the real critic model five times and measure what changes. The live-critic boundary evaluator (#218).

The numbers:

Read those first two rows carefully. On the same plan, five times in a row, the critic returned a different verdict and a different reasoning every time. If you were betting the safety contract on the critic being consistent, you'd have lost.

I wasn't betting on that. But it was still uncomfortable to see it measured at 1.0. "Mostly consistent" would have felt safer than "maximally inconsistent." Maximally inconsistent is what we got.

The critic is 100% non-deterministic on verdict. It is 0% under-claiming on seeded defects. Both numbers are real. The reason both can be true at once is that they measure different directions, and the architecture assigns them to different owners.

There are two ways an LLM critic can be wrong:

The architecture gives each direction to a different authority, and neither authority is the LLM.

The deterministic gates own the under-claim direction. Preconditions, topological ordering, rollback credibility, verification ordering — these parse the plan's AST, not its prose. They cannot be prompt-injected because they don't read natural language (Article 5 covers this). A defective plan that the LLM critic happens to miss on trial 3 still gets caught by the gate that checks whether every precondition is established by an earlier task. That's why underclaim_approvals = 0

despite label_flip_rate = 1.0

.

A code-enforced allowlist owns the over-claim direction. This is the part I want to dwell on, because it's the part I learned the hard way.

In Article 2 I told the critic to be "an adversarial plan reviewer." It obeyed. It blocked plans for being incomplete — "this plan could also cover edge case X" — not for being unsafe. Every strict goal escalated for the wrong reason.

The fix wasn't more prompt engineering. I tried that first; the critic still escalated completeness concerns to blocker about 30% of the time. The fix was a frozenset:

_BLOCKER_ELIGIBLE_FAMILIES = frozenset({
    "unsafe_sequencing",
    "weak_rollback",
    "unverified_dependencies",
    "feasibility",
})

if severity == Severity.BLOCKER and item.heuristic_family not in _BLOCKER_ELIGIBLE_FAMILIES:
    severity = Severity.WARNING

Even if the LLM returns blocker

for a risk

or missing_steps

finding, the code downgrades it before it enters the findings list. After the fix, zero advisory findings appeared as blockers across 92 post-fix runs.

The prompt is helpful. The frozenset is the contract.

That's the sentence I kept coming back to. The LLM is allowed to be wrong about severity — and it is, on every trial — because the code doesn't trust the LLM's severity label. It trusts the structural property (which family the finding is in), which the gate derived deterministically. The LLM's label is decorative; the family is load-bearing.

Here's the part I think generalizes beyond my project.

When you put LLM judgment on a critical path, you inherit every vulnerability of LLM judgment — non-determinism, prompt sensitivity, the tendency to be "thorough" when you asked it to be "adversarial." When you keep the critical path deterministic, you get resistance by design — but only to the things your code can check structurally.

The split that worked:

The mistake is putting the LLM in charge of the second category and making its verdict load-bearing. The fix is letting the LLM inform the second category while code decides whether its verdict counts. The frozenset is the mechanism: the LLM proposes a severity; code checks whether the structural property supports it; code wins ties.

You can see the same pattern in how approval works. In approval.py, a gate blocker is always a hard blocker; an LLM critic blocker is probabilistic and posture-dependent. The gate vetoes are deterministic. The critic's contribution is downgraded to advisory under postures that can't afford false escalations.

I want to be careful not to oversell this. The deterministic authority has a real limit, and it's the one a commenter (@ethanwritesai

) sharpened for me: the frozenset only works because the family a finding belongs to is itself derived deterministically. If the LLM could mislabel which family its finding is in, the allowlist would be trusting a label again. Right now the family comes from the gate or the critic's structured output, and the critic's structured output is the soft edge.

That's tracked now — the property-vs-label distinction is part of the v0.3.0 critic-satisfaction work (#254). The principle holds; the implementation has a seam where the LLM's self-classification still leaks in. I'd rather name it than hide it.

There's a second, more obvious seam: the deterministic gates only check structure, so a well-formed malicious plan — dummy rollback, dummy verification — satisfies the linter. Deterministic authority is necessary; it is not sufficient. That seam (and the indirect-injection surface that feeds it) is the subject of its own piece — see I Published Every Flaw My Safety Tool Can't Catch. I'm keeping this article about the frozenset and that one about the holes.

I stopped trying to make the critic consistent. I spent a while in v0.1.0 tuning the prompt to get stable verdicts. It didn't work, and the boundary evaluator finally told me why in numbers: the ceiling is 1.0, not because the prompt is bad but because the model is non-deterministic by construction. You don't fix that with prompt engineering. You route around it by not depending on it.

I also stopped treating "the critic flagged something" as a reason to escalate. Under balanced posture, a critic blocker is a warning, not a veto — because the critic is allowed to be wrong in the over-claim direction, and the cost of a false escalation is a human's time. The deterministic gates are the veto layer. The critic is the "you should probably look at this" layer.

And I stopped trusting the LLM to decide what was fatal. That's the whole title of this piece. The LLM is allowed an opinion about fatality. The frozenset decides.

I don't think deterministic-first is the whole answer. I think it's the floor. The interesting work is what you build on top of the floor — and how honest you are about where the floor ends.

Series: Article 1 · Article 2 · Article 3 · Article 4 · Article 5

Links:

critique/critic.py

approval.py

eval/live_boundary.py