Two Projects, One Problem — What PlannerCritic and AdversarialDebate Each Got Wrong

A developer built two LLM-based systems, PlannerCritic and AdversarialDebate, to detect when an LLM's judgment is wrong. Both systems proved reliable but shared a blind spot: their safety metrics improved when a gate or protocol silently stopped firing, masking underlying problems. The developer credits a community reader's comment for reframing the issue.

I built two systems this year that try to solve the same problem from opposite directions. PlannerCritic https://github.com/deghosal-2026/planner-critic-engine puts an LLM critic in a loop with a deterministic gate layer. The gates own the safety contract. The critic is advisory. A planner decomposes a goal into a typed plan. A critic audits every subtask. The plan is revised until approval — or escalated to a human. AdversarialDebate https://github.com/deghosal-2026/adversarial-debate puts two LLMs in a structured debate. The protocol owns the independence. The models are the evidence. Two reviewers analyze the same artifact without seeing each other's answers, debate their conclusions point by point, and produce either a converged decision — or a structured disagreement report that preserves the dissent. Both systems are about the same question: how do you know when an LLM's judgment is wrong? I thought the answer was architecture. Build the right structure, put code where the model can't be trusted, and the system becomes reliable. That was the thesis of both projects. PlannerCritic bet on code. AdversarialDebate bet on structure. I was half right. The architecture worked. The systems are reliable. But both had the same blind spot, and it took me months to see it — because the blind spot made the metrics look better , not worse. PlannerCritic's bet was code over model . The decision path is a frozenset of blocker-eligible families — unsafe sequencing , weak rollback , unverified dependencies , feasibility . The LLM critic can label a finding as blocker , but if the family isn't in the frozenset, the system downgrades it to a warning. The model's severity label is decorative. The family is load-bearing. AdversarialDebate's bet was structure over consensus . A revelation gate ensures neither reviewer sees the other's output before committing. The debate protocol requires point-by-point responses — CONCEDED , REBUTTED , or CARRIED . The convergence score measures movement, not agreement. Two models that agree without engaging get a low score. Two models that genuinely challenge each other get a high one. On paper, those bets are compatible. Code guards against structural under-claim. Debate guards against shared hallucination. Together, they cover more ground than either alone. But each bet had a failure mode that the other system's design would have caught immediately. And neither system caught its own — because the failure made the dashboard look like things were improving. In PlannerCritic, the deterministic gates own safety. They parse the plan's structure — ordering, preconditions, rollback, verification — and they block anything that doesn't meet the contract. By v0.2.2, the blocker counts across the full 183-goal sweep were dominated by structural families: unsafe sequencing at 226 blockers, unverified dependencies at 185, weak rollback at 86. Those numbers felt like evidence that the gates were doing their job. Then a community reader named Artjoms Stukans left a comment that reframed how I thought about the entire system: "If one blocker class stops firing after some refactor, your numbers only look better. 226 becomes 40 and that reads like plans got safer." He described a Kubernetes incident where four releases in a row never actually ran. An old ReplicaSet kept one pod Running. Every health check passed. Every smoke test was green. Nothing said a word — because the pod was still in a "healthy" state, even though the deployment was broken. That is exactly the shape of PlannerCritic's problem. A gate that silently stops firing produces better metrics. The blocker count drops. The dashboard reads as improved safety. The variance signal — label flip rate , evidence drift rate — the very metrics the project uses to surface critic problems — vanishes exactly where the safety contract moved. The critic is watched. The gates are trusted. Trusted layers don't advertise their own failures. That's the trap. AdversarialDebate already had the pattern for catching this. Each debate produces a transcript, and the capitulation detector reads that transcript for the shape of failure instead of trusting the aggregate convergence score. It doesn't ask "did the debate converge?" It asks "did one side concede everything in round 1 without a single rebuttal?" Those are different questions, and only the second one catches the failure. PlannerCritic had no equivalent. It tested the gates at build time. It did not monitor them at runtime. The fix was the Gate Canary https://github.com/deghosal-2026/planner-critic-engine/issues/278 — ten fixture pairs, one per gate class, each with a known-good plan and a known-bad plan. The CLI command plancritic gates canary --check runs them in under a second at zero LLM cost and exits 1 if any gate stopped firing on its bad plan. It's a dumb check. It's a cheap check. It catches exactly the failure mode that would have been invisible in every other metric the system produces. It shipped in v0.2.3 https://github.com/deghosal-2026/planner-critic-engine/blob/main/docs/reference/release-notes-v0.2.3.md . In AdversarialDebate, the strongest pair by convergence score was DeepSeek + Mistral: 0.982 average convergence, 97% verdict rate, 2,352 concessions across 41 debates. Every metric in the field test report https://github.com/deghosal-2026/adversarial-debate/blob/main/docs/field-test/v0.2.2/FIELD TEST REPORT.md pointed to this pair as the best production default. Then I read the transcripts. 65% of those debates were capitulation cascades — one side conceding every claim in round 1 with zero rebuttals. The pair converged beautifully. It was almost useless for adversarial review. The metrics saw convergence. The transcripts saw surrender. PlannerCritic would have caught this. Its deterministic gates cannot be talked into approving a plan. A capitulation cascade that passes every aggregate check looks structurally different to a gate that inspects individual preconditions and rollback steps. The debate metrics measured whether the models agreed. PlannerCritic's gates would have measured whether the plan was safe. Those are not the same question, and only the second one catches the failure. AdversarialDebate had no deterministic floor. The protocol was fine; the pair was bad; the metrics could not distinguish the two. The fix was the capitulation detector — transcript-level inspection independent of convergence scores. But it was a detection mechanism added after the first surprise, not a health check designed before it. Then v0.2.2 went further. The noise-floor baseline https://github.com/deghosal-2026/adversarial-debate/issues/165 gave every metric a known confidence interval via bootstrap resampling — the GPT+Mistral vs DeepSeek+Mistral gap 0.536 vs 0.572 is 1.8 sigma, real but narrow. The permutation control https://github.com/deghosal-2026/adversarial-debate/issues/164 shuffled claim-to-ground-truth pairings 500 times to build a null distribution — the LLM judge's 87.4% match rate sits 77.8 standard deviations above the vocabulary floor. The shared RLHF priors https://github.com/deghosal-2026/adversarial-debate/issues/166 design note documented that the Mistral effect might be driven as much by non-Mistral models rubber-stamping each other as by Mistral's unique training. Each fix was a separate measurement layer, added because the previous layer couldn't see the failure. Both failures have the same anatomy: A layer you trusted failed silently, and the system interpreted the silence as improvement. The fix in both cases was the same pattern: a secondary check that inspects the shape of the output , not just its aggregate value. PlannerCritic needed the gate canary. AdversarialDebate needed the capitulation detector. Neither check is expensive. Neither is clever. Both are the kind of thing you add after the first time the metrics lie to you. Here's what I find most uncomfortable about this: I built both systems specifically because I didn't trust LLM judgment. I moved the safety contract into deterministic code. I built isolation protocols. I measured non-determinism. And then I trusted the deterministic layer so completely that I stopped watching it. The monitoring I built was for the LLM — label flip rate , evidence drift rate , capitulation cascade . The layer that actually held the contract — the gates, the debate protocol — had no monitoring at all. A deterministic safety layer does not remove the monitoring problem. It moves it. Before the gate canary, PlannerCritic monitored the critic's non-determinism. After the fix, monitoring also had to cover the gates themselves — not their aggregate output, but their continued ability to fire. Before the capitulation detector, AdversarialDebate monitored protocol compliance — rounds completed, claims resolved. After the fix, monitoring also had to cover whether the protocol was producing genuine engagement or procedural surrender. The monitoring target shifts when you trust the deterministic layer. That is the blind spot both projects hit. And it's the blind spot I think most agent infrastructure has right now — we're all building deterministic guardrails and then walking away. Both projects shipped their fixes in late August 2026. The fixes are real, tested, and in production. PlannerCritic v0.2.3 release notes https://github.com/deghosal-2026/planner-critic-engine/blob/main/docs/reference/release-notes-v0.2.3.md , field test report https://github.com/deghosal-2026/planner-critic-engine/blob/main/docs/field-test/v0.2.3/FIELD TEST REPORT.md : PermissionError now fires from CLI, HTTP, and MCP — not just tests AdversarialDebate v0.2.2 release notes https://github.com/deghosal-2026/adversarial-debate/blob/main/docs/reference/release-notes-v0.2.2.md , field test report https://github.com/deghosal-2026/adversarial-debate/blob/main/docs/field-test/v0.2.2/FIELD TEST REPORT.md : Both projects now have measurement infrastructure that catches the failure class this article describes. But the infrastructure was built after the failure was found — by reading transcripts, by community comments, by running the benchmark and getting a number I didn't want. If I were building a third system — or advising someone building their first — I would start with what both projects missed. A cross-system invariant checker. Not a feature. Not a gate. A dumb, cheap, separate utility that runs after every eval and asserts that the output was produced under honest conditions. It doesn't need to understand the domain. It needs to know the shape of the output and check whether that shape is still the right shape. For PlannerCritic, that means: each gate class still fires on its canary fixture. The redaction layer didn't corrupt a number. The decision context wasn't "unknown." For AdversarialDebate, that means: the noise floor was measured and the reported difference exceeds it. The permutation control null distribution is stable. No debate exceeded the capitulation threshold. The same checker. Different project configs. A shared commitment to not trusting your own outputs. The two projects were built months apart, on different architectures, for different use cases. PlannerCritic is about planning safety — does the plan survive structural inspection? AdversarialDebate is about review quality — does the second opinion actually challenge the first? But they converged on the same engineering lesson, and it's the one I'd put on a wall: The layer you trust the most is the layer you are most likely to stop watching. The fix is to watch it anyway — with a separate, cheaper, dumber check that exists only to tell you whether the expensive layer is still alive. Not because the expensive layer is unreliable. Because you are unreliable when the metrics are green and the dashboard says "improved." That is what the gate canary does for PlannerCritic. That is what the capitulation detector does for AdversarialDebate. And that is what I would build first, not last, in the next system — before the first field test, before the first release, before the first time the metrics lie to me and I don't notice. Because the first time is always the one that ships. Previous AdversarialDebate articles Links