The Planner Made the Same 3 Mistakes Every Time. A Bigger Model Didn't Fix It.

An engineer building PlannerCritic, an open-source engine where one LLM writes a plan and a second reviews it, found that the planner consistently makes three structural mistakes—unverified dependencies, unsafe sequencing, and weak rollback—across 132 concrete blockers in a 157-goal field test. Upgrading to GPT-4o did not fix the pattern, leading the developer to conclude that deterministic validation, not larger models, is needed to address the planning-structure problem.

This is article 3 in a series about building PlannerCritic , an open-source engine where one LLM writes a plan and a second LLM reviews it. Article 1 covers the 157-goal field test. Article 2 is about the critic severity bug. This one is about the most uncomfortable thing the field test revealed: the planner has a structural problem that no model upgrade fixes.132 concrete blockers across 63 strict goals. Three defect families. I tried gpt-4o. Same pattern. The fix is deterministic validation, not more parameters. By the 10th strict goal, I noticed it. By the 50th, I could predict the failure before the critic printed it. By the 100th, I stopped being surprised and started being annoyed. The planner kept making the same three mistakes. Not occasionally. Not randomly. Every single strict goal that failed did so because of one of three defect families. Unverified dependencies 57 blockers : The plan declares a precondition that no earlier task establishes. The planner knows what should be true. It doesn't arrange the steps to make it true. Real example — from ai-03-model-serving-migration : BLOCKER unverified dependencies — task=cutover traffic 100 "Cutover traffic from SageMaker to vLLM 100% verification of latency SLO is dependent on prior traffic cutover stages being established but lacks clear confirmation of stability before proceeding." The plan says "cutover 100% of traffic" but no earlier task verifies that the 10% and 50% stages were stable. Unsafe sequencing 46 blockers : Tasks are ordered before their hard prerequisites. The cutover runs before the verification. The backup completes after the migration. Real example — from ai-02-embedding-index-migration : BLOCKER unsafe sequencing — task=backfill vectors "Backfill operation cannot proceed until the index is verified for quality; it is ordered incorrectly in the sequence." The plan puts the backfill before the quality check that should gate it. Weak rollback 18 blockers : High-blast-radius steps lack rollback. The planner includes rollback on routine steps but omits it on cutover, teardown, and failback. Real example — from db-10-multi-tenant-split : BLOCKER weak rollback — task=dual write setup "The dual-write setup task's rollback only switches to single-write mode without addressing potential inconsistencies during the transition." The rollback exists but it is not credible. Switching back to single-write does not undo the data inconsistencies that dual-write may have introduced. The natural instinct: use gpt-4o. I kept waiting for it to bail me out. I tested it both ways — gpt-4o planner with mini critic, and gpt-4o for both roles. Same defect pattern. Better prose. Same structural mistakes. Unverified dependencies. Unsafe sequencing. Weak rollback. That was the moment I stopped blaming the model size. The planner wasn't dumb. That was the annoying part. It was plausible. It knew the right steps. It just couldn't close the dependency graph or enforce the ordering. I did not have a smaller-model problem. I had a planning-structure problem. The revision loop is designed to converge. The critic reports blockers. The planner revises. But the planner tends to fix one blocker and introduce another. It reshuffles task order without closing the dependency gap. It adds rollback to the wrong task. After 2 revisions — the median across 33 strict goals — the planner stops making meaningful changes. The convergence detector fires. The engine escalates. The loop works as designed. The planner is the bottleneck. I kept expecting the revision loop to converge. It didn't, because the planner can't fix a structural problem by rewriting the prose. Worth being clear about what failed here. The critic reliably found the same blockers across revisions — the failure was the planner's inability to structurally repair, not the critic's judgment. That's a different defect from the one in Article 2, where the critic's severity calibration was the problem. The highest-leverage fix is a precondition closer : a deterministic linter that runs after the planner produces a draft and verifies that every precondition is actually established by an earlier task. If a task says "requires replica verified," there must be a prior task that produces it. This single pass would eliminate 64 of 132 blockers 48% without asking the LLM to get smarter. The remaining blockers — unsafe sequencing and weak rollback — need either better prompt engineering, additional deterministic validation, or genuinely better reasoning about ordering and risk. This is not just my observation. The academic literature converges on the same finding. The "Why Reasoning Fails to Plan" paper arXiv 2601.22311 shows that LLM agents select actions based on local evaluation without considering future consequences. In knowledge graph traversals, single-step greedy policies select myopic traps more than 55% of the time. The authors prove step-wise reasoning is provably insufficient for long-horizon planning. The PlanGenLLMs survey arXiv 2502.11221 evaluates LLM planning across four criteria — completeness, executability, optimality, representation. LLMs consistently fail at ensuring plans are executable: preconditions not met, steps out of order. The field is converging on hybrid approaches — LLM plus deterministic validation plus classical planning techniques. Not LLM alone. If you are building agents that plan over multiple steps: Article 3 of 5 in the PlannerCritic series. Series: Article 1: "I Ran 157 Agent Plans Against a Real LLM" https://dev.to/debashish ghosal/i-ran-157-agent-plans-against-a-real-llm-the-problem-wasnt-execution-it-was-planning-163j · Article 2: "I Told My LLM Critic to Be Adversarial" https://dev.to/debashish ghosal/i-told-my-llm-critic-to-be-adversarial-it-started-blocking-plans-for-being-not-thorough-enough-2cd8 · Article 4: "The Field Test Found 10 Issues" · Article 5: "I Tried to Prompt-Inject My Own Engine" Links: pip install planner-critic