This is article 4 in a series about building
[PlannerCritic], an open-source engine where one LLM writes a plan and a second LLM reviews it.[Article 1]covers the 157-goal v0.1.0 field test.[Article 2]is about the critic severity bug.[Article 3]is about the planner capability gap. This one is a practical guide to field testing agent systems — from 157 goals at $0.30 in v0.1.0 to 170 goals at $0.49 in v0.2.1, and what 10→0 issues across three releases teaches about release engineering.
Updated for v0.2.1:The original article covered the v0.1.0 field test — 157 goals, $0.30, 10 issues found. Since then, the engine shipped v0.2.0 (170 goals, 31 code-review bugs fixed, 0 field-test-found issues) and v0.2.1 (170 goals, 10 more code-review bugs fixed, 0 field-test-found issues, live-critic boundary evaluator, operational benchmark). This update tells the full arc: from a diagnostic tool that found 10 issues to a regression gate that finds 0 — and why 0 is the hardest result to earn.If a field test returns 0 failures, your first instinct should be distrust. In v0.1.0, 0 failures meant our harness was silently broken. In v0.2.1, 0 field-test issues meant something very different: code review caught all 41 bugs before the LLM ran, turning the field test from a diagnostic sweep into an immutable regression gate.
Unit tests test what you think to test. They use hand-crafted inputs that match your assumptions. They pass even when the system is wrong in ways you didn't anticipate.
I know this because 57 of 65 assertion files were in the wrong format and the harness didn't even notice. It just quietly returned 0/0 results. No crash. No error. Just silence.
I needed a field test that would run real goals against a real LLM and tell me what actually broke.
The first field test was a diagnostic. It found 10 issues across 157 goals in 35 domains for $0.30 in 60 minutes. Only 1 was a traditional failure. The rest were design problems, prompt gaps, and harness bugs that would have shipped silently.
1 True Failure: The planner prompt didn't explain the branches schema. The LLM responded with kind: "rollback"
and arrays of task objects where strings were required.
4 Design Issues:
established_by
expected a task ID or env:
prefix; the LLM wrote bare fact names. Unit tests were green.run_budget()
takes 4 arguments but dispatch passed 5. Budget and replan dimensions showed 0/0.2 Harness Bugs:
`trace.get("status")`
instead of `trace["result"]["status"]`
. Five PASS scenarios were misreported as FAIL.1 Model Limitation: Local models (Qwen3.5-4B, Qwen3.5-9B) couldn't produce structured JSON.
2 Fundamental Properties:
revision_cap=4
all escalated; the critic found different blockers on each revision.$0.30 in LLM API calls. 60 minutes of wall-clock time. Cheaper than debugging one production incident caused by a bad plan.
The v0.2.0 field test was fundamentally different. v0.1.0 was a diagnostic tool that found 10 issues in a greenfield engine. v0.2.0 was a validation tool that confirmed 31 bug fixes + 5 new enterprise domain packs + 6 new safety mechanisms all work correctly.
The field test itself found zero new issues — the code review found all 31 bugs before the field test ran.
The v0.2.0 field-test program was substantially stronger than v0.1.0's, and that directly produced a higher-quality release:
| Metric | Result |
|---|---|
| Balanced goals approved | 73/73 (100%) |
| Strict goals escalated | 97/97 (100%) |
| Adversarial goals escalated | 8/8 (100%) |
| True failures | 0 |
| Deterministic gate passes | 170/170 (100%) |
| Security oracle | 7/7 correct, 35/35 flawed blocked, 21 traps |
| Scorecard A | PASS |
| Code-review bugs fixed | 31 |
| Field-test-found bugs | 0 | v0.2.1 is a patch release. No new goals, no new domains, no schema changes. The field test is a regression sweep — re-run the same 170 goals and diff every verdict against the published v0.2.0 results.
The code review (#222) found 10 bugs in the M11 hardening diff. The field test validated the fixes. 30 verdict deltas vs v0.2.0 — all attributable, zero unexplained.
| Issue | Finding | Why It Matters |
|---|---|---|
| #232 | Histogram cycling detector dead under default config | A safety signal shipped but was unreachable unless you manually raised revision_cap above 3. The detector was there. It just never fired. |
| #233 | Rollback-credible gate emitted bare task id as message | The blocker said "t2" . No description, no suggested fix. An operator would have no idea what to do. |
| #234 | Finding ids collided — distinct defects merged silently | Two different producers offending against the same consumer produced byte-identical finding ids. The escalation system's {f.id: f} merge collapsed one defect. |
| #235 | Live-critic runner lost entire run on mid-trial exception | One transient LLM timeout discarded 60 completed audits. No partial report, no error record. |
| #236 | Adapter import test gutted to pass |
|
A test named test_all_adapters_importable that asserted nothing. It would pass forever, even if every adapter was broken. |
||
| #237 | Suggested-fix printed task id where group name belonged | "Move out of parallel group 'deploy'" — but 'deploy' is a task id, not a group name. |
| #238 | Approval authority enforcement not wired to any shipped surface | Test-proven but unreachable from CLI/HTTP/MCP. Documented as F-14, deferred to v0.3.0. |
| #239 | Evidence-drift metric pooled explanations across trials | A deterministic 2-finding critic scored drift=1.0 because the metric counted intra-trial variety as cross-trial drift. |
| #240 | ApprovalGate stamped ambient goal posture, not contract | A STRICT contract bound onto a BALANCED goal produced an ApprovedPlan that said BALANCED. Downstream audits read the wrong regime. |
| #241 | Content hash preserved criteria insertion order | Same rules in different order hashed differently. Latent today (single-criterion binding), but breaks equivalence once multi-criterion contracts exist. |
Every fix shipped with a regression test written RED-first (verified failing on pre-fix code, then made green by the fix).
v0.2.1 added a new test that v0.1.0 and v0.2.0 didn't have: send the #171 boundary-case corpus through the real critic model × 5 trials and measure what happens when you ask the same question 5 times.
The critic is 100% non-deterministic — and that's fine.gpt-4o-mini produces a different verdict and different explanation on every trial of identical input (label_flip_rate=1.0, evidence_drift_rate=1.0). Yet it
never under-claims a seeded defect(family_migration_rate=0.0, underclaim_approvals=0). The deterministic gates are the security authority — the LLM critic's non-determinism is safe because it can only add findings, never suppress gate blockers.
| Metric | Value | What It Means |
|---|---|---|
| label_flip_rate | 1.000 | The critic changes its verdict on every trial of identical input |
| evidence_drift_rate | 1.000 | The critic invents different explanations every trial |
| family_migration_rate | 0.000 | No seeded defect landed in an advisory family |
| underclaim_approvals | 0 | No defective plan got zero blockers |
This is the finding I didn't expect: the critic is maximally non-deterministic, and it doesn't matter. The safety contract doesn't depend on the critic being consistent — it depends on the critic always finding something on defective plans. And it does.
Takeaway:Deterministic gates own the under-claim direction (preventing bad plans from slipping through), while code-enforced severity allowlists own the over-claim direction. The LLM critic can be 100% non-deterministic and still completely safe.
v0.2.1 added the before/after numbers the community asked for:
| Metric | Value |
|---|---|
| Latency (approved) p50 | 13.86s |
| Latency (escalated) p50 | 27.82s |
| Mean blockers per goal | 2.58 | | Mean advisories per goal | 1.86 | | Escalation decisions per 100 goals | 58.0 | | Mean LLM calls per goal | 1.4 | | Median revisions to resolution | 1.0 |
The median revisions to resolution is 1.0 — most goals resolve in a single revision. The deterministic precondition closer and topological auto-repair are doing their job: they fix ordering and dependency defects without calling the LLM at all.
| Metric | v0.2.0 | v0.2.1 | Delta |
|---|---|---|---|
| Goals swept | 170/170 | 170/170 | same |
| Balanced approved | 73/73 (100%) | 73/73 (100%) | same |
| Strict escalated | 97/97 (100%) | 96/97 (99%) | 1 transient provider error |
| Adversarial aborted | 8/8 | 8/8 | same |
| Verdict deltas vs prior | — | 30 | all attributable |
| Deterministic tests | 90 | 1295 | +1205 (contract tests for every new gate, evaluator, and schema) |
| Benchmarks | 3 | 3 | +operational, +boundary |
| Code-review bugs fixed | 31 | 10 | — |
| Field-test-found bugs | 0 | 0 | same |
plan_oscillation_detected |
0 | 5 | #152 now fires |
| Coverage | 91.62% | 91.58% | -0.04% (accepted) | 30 goals changed their verdict or reason code between v0.2.0 and v0.2.1. Every single one is attributable:
converged_stalled
and revision_cap_reached
depending on which specific blockers the critic found this time.plan_oscillation_detected
in v0.2.1 (0 in v0.2.0). The #232 fix made the detector reachable under default config. It detects cycling earlier and terminates the loop sooner — saving LLM calls.mch-04-blast-radius
hit planning_unavailable
— OpenRouter returned an error. The engine correctly failed closed (escalated as error, not approved).Zero unexplained deltas. No delta is attributable to a code-review fix changing engine behavior — the fixes improved internal consistency (gate ids, message quality, fault isolation) without altering the approve/escalate decision logic.
The arc across three releases tells a story about how field testing evolves:
| Release | Goals | Cost | Issues Found by Field Test | Issues Found by Code Review |
|---|---|---|---|---|
| v0.1.0 | 157 | $0.30 | 10 | 0 |
| v0.2.0 | 170 | $0.40 | 0 | 31 |
| v0.2.1 | 170 | $0.49 | 0 | 10 |
The field test went from a diagnostic (found 10 issues in a greenfield engine) to a validation tool (confirmed 31 fixes in v0.2.0) to a regression gate (confirmed 10 more fixes, diffed against the published baseline, zero unexplained deltas).
The cost went from $0.30 to $0.49. The value went from finding bugs to proving their absence.
In v0.1.0, only 1 of 10 issues was a traditional failure. The rest were design problems that would have shipped silently. The harness quietly lying about assertion results was worse than a crash would have been.
Lesson: A field test harness that executes zero assertions for a module must fail hard — 0/0 is an error state, not a pass.
Every issue that involved the LLM behaving unexpectedly was invisible in unit tests with hand-crafted inputs. The preconditions gate bug was hiding behind green tests.
Lesson: Unit tests can't replace field tests. They test different things.
v0.1.0 used the field test to find 10 issues (~$0.30 + 60 min). v0.2.0 used code review to find 31 bugs ($0 + 2 hours). v0.2.1 found 10 more ($0 + 1 hour). The field test found 0 in both cases.
Lesson: For a mature engine with a proven corpus, code review before field test is the higher-leverage activity. The field test validates; the code review diagnoses.
30 verdict deltas between v0.2.0 and v0.2.1 are all attributable to gpt-4o-mini producing different findings across runs. The #218 live-critic boundary run measured this directly: label_flip_rate=1.0, evidence_drift_rate=1.0 — the critic changes its verdict and explanation on every trial of identical input. Yet family_migration_rate=0 and underclaim_approvals=0 — it never under-claims a seeded defect.
Lesson: Deterministic gates own the under-claim direction (preventing bad plans from slipping through), while code-enforced severity allowlists own the over-claim direction. The LLM critic can be 100% non-deterministic and still completely safe.
3 goals now escalate with plan_oscillation_detected
instead of revision_cap_reached
. The signal detects cycling earlier and terminates the loop sooner.
Lesson: The oscillation signal fires in practice (3/97 strict goals) and saves LLM calls.
1 goal hit planning_unavailable
— OpenRouter returned an error. The engine correctly failed closed (escalated as error, not approved).
Lesson: Transient LLM provider errors are not engine defects; the engine fails closed.
Latency (p50 approved=13.86s), reviewer burden (2.58 blockers/goal), operator workload (58 decisions/100 goals). Median revisions to resolution = 1.0. Lesson: Operational baselines enable before/after comparison — the downstream-error-rate metric requires partner runner integration (deferred to v0.3.0).
$0.49 for 170 goals + 60 boundary audits. Cheaper than a single developer-hour. There is no excuse not to field test your agent system.
Lesson: The field test should run on every release. The cost is negligible.
The deterministic gates run on every commit for free. The 1295 deterministic tests run in 4.7 seconds. The LLM field test runs on release for $0.49. Three tiers, each catching different things.
Lesson: Unit tests can't replace the deterministic gate suite. The deterministic suite can't replace the field test. The field test can't replace code review. All three are needed.
v0.3.0 will:
approving_authority
through CLI/HTTP/MCP (F-14)Article 4 of 5 in the PlannerCritic series.
Series: Article 1: "I Ran 157 Agent Plans Against a Real LLM" · Article 2: "I Told My LLM Critic to Be Adversarial" · Article 3: "The Planner Made the Same 3 Mistakes" · Article 5: "I Tried to Prompt-Inject My Own Engine"
Links:
pip install planner-critic
—