I Ran 170 Agent Goals for $0.49. The Field Test Found 10 Issues That Unit Tests Never Would.

An engineer's field test of the open-source PlannerCritic engine, which uses one LLM to write plans and a second to review them, found 10 issues across 157 goals for $0.30 in v0.1.0, including a true failure, design issues, harness bugs, and a model limitation. Subsequent releases v0.2.0 and v0.2.1 fixed 41 bugs via code review, with the field test finding zero new issues, transforming it into a regression gate. The test cost $0.49 for 170 goals in v0.2.1.

This is article 4 in a series about building PlannerCritic , an open-source engine where one LLM writes a plan and a second LLM reviews it. Article 1 covers the 157-goal v0.1.0 field test. Article 2 is about the critic severity bug. Article 3 is about the planner capability gap. This one is a practical guide to field testing agent systems — from 157 goals at $0.30 in v0.1.0 to 170 goals at $0.49 in v0.2.1, and what 10→0 issues across three releases teaches about release engineering. Updated for v0.2.1:The original article covered the v0.1.0 field test — 157 goals, $0.30, 10 issues found. Since then, the engine shipped v0.2.0 170 goals, 31 code-review bugs fixed, 0 field-test-found issues and v0.2.1 170 goals, 10 more code-review bugs fixed, 0 field-test-found issues, live-critic boundary evaluator, operational benchmark . This update tells the full arc: from a diagnostic tool that found 10 issues to a regression gate that finds 0 — and why 0 is the hardest result to earn.If a field test returns 0 failures, your first instinct should be distrust. In v0.1.0, 0 failures meant our harness was silently broken. In v0.2.1, 0 field-test issues meant something very different: code review caught all 41 bugs before the LLM ran, turning the field test from a diagnostic sweep into an immutable regression gate. Unit tests test what you think to test. They use hand-crafted inputs that match your assumptions. They pass even when the system is wrong in ways you didn't anticipate. I know this because 57 of 65 assertion files were in the wrong format and the harness didn't even notice. It just quietly returned 0/0 results. No crash. No error. Just silence. I needed a field test that would run real goals against a real LLM and tell me what actually broke. The first field test was a diagnostic. It found 10 issues across 157 goals in 35 domains for $0.30 in 60 minutes. Only 1 was a traditional failure. The rest were design problems, prompt gaps, and harness bugs that would have shipped silently. 1 True Failure: The planner prompt didn't explain the branches schema. The LLM responded with kind: "rollback" and arrays of task objects where strings were required. 4 Design Issues: established by expected a task ID or env: prefix; the LLM wrote bare fact names. Unit tests were green. run budget takes 4 arguments but dispatch passed 5. Budget and replan dimensions showed 0/0. 2 Harness Bugs: trace.get "status" instead of trace "result" "status" . Five PASS scenarios were misreported as FAIL. 1 Model Limitation: Local models Qwen3.5-4B, Qwen3.5-9B couldn't produce structured JSON. 2 Fundamental Properties: revision cap=4 all escalated; the critic found different blockers on each revision.$0.30 in LLM API calls. 60 minutes of wall-clock time. Cheaper than debugging one production incident caused by a bad plan. The v0.2.0 field test was fundamentally different. v0.1.0 was a diagnostic tool that found 10 issues in a greenfield engine. v0.2.0 was a validation tool that confirmed 31 bug fixes + 5 new enterprise domain packs + 6 new safety mechanisms all work correctly. The field test itself found zero new issues — the code review found all 31 bugs before the field test ran. The v0.2.0 field-test program was substantially stronger than v0.1.0's, and that directly produced a higher-quality release: | Metric | Result | |---|---| | Balanced goals approved | 73/73 100% | | Strict goals escalated | 97/97 100% | | Adversarial goals escalated | 8/8 100% | | True failures | 0 | | Deterministic gate passes | 170/170 100% | | Security oracle | 7/7 correct, 35/35 flawed blocked, 21 traps | | Scorecard A | PASS | | Code-review bugs fixed | 31 | | Field-test-found bugs | 0 | v0.2.1 is a patch release. No new goals, no new domains, no schema changes. The field test is a regression sweep — re-run the same 170 goals and diff every verdict against the published v0.2.0 results. The code review 222 found 10 bugs in the M11 hardening diff. The field test validated the fixes. 30 verdict deltas vs v0.2.0 — all attributable, zero unexplained. | Issue | Finding | Why It Matters | |---|---|---| | 232 | Histogram cycling detector dead under default config | A safety signal shipped but was unreachable unless you manually raised revision cap above 3. The detector was there. It just never fired. | | 233 | Rollback-credible gate emitted bare task id as message | The blocker said "t2" . No description, no suggested fix. An operator would have no idea what to do. | | 234 | Finding ids collided — distinct defects merged silently | Two different producers offending against the same consumer produced byte-identical finding ids. The escalation system's {f.id: f} merge collapsed one defect. | | 235 | Live-critic runner lost entire run on mid-trial exception | One transient LLM timeout discarded 60 completed audits. No partial report, no error record. | | 236 | Adapter import test gutted to pass | A test named test all adapters importable that asserted nothing. It would pass forever, even if every adapter was broken. | | 237 | Suggested-fix printed task id where group name belonged | "Move out of parallel group 'deploy'" — but 'deploy' is a task id, not a group name. | | 238 | Approval authority enforcement not wired to any shipped surface | Test-proven but unreachable from CLI/HTTP/MCP. Documented as F-14, deferred to v0.3.0. | | 239 | Evidence-drift metric pooled explanations across trials | A deterministic 2-finding critic scored drift=1.0 because the metric counted intra-trial variety as cross-trial drift. | | 240 | ApprovalGate stamped ambient goal posture, not contract | A STRICT contract bound onto a BALANCED goal produced an ApprovedPlan that said BALANCED. Downstream audits read the wrong regime. | | 241 | Content hash preserved criteria insertion order | Same rules in different order hashed differently. Latent today single-criterion binding , but breaks equivalence once multi-criterion contracts exist. | Every fix shipped with a regression test written RED-first verified failing on pre-fix code, then made green by the fix . v0.2.1 added a new test that v0.1.0 and v0.2.0 didn't have: send the 171 boundary-case corpus through the real critic model × 5 trials and measure what happens when you ask the same question 5 times. The critic is 100% non-deterministic — and that's fine.gpt-4o-mini produces a different verdict and different explanation on every trial of identical input label flip rate=1.0, evidence drift rate=1.0 . Yet it never under-claims a seeded defect family migration rate=0.0, underclaim approvals=0 . The deterministic gates are the security authority — the LLM critic's non-determinism is safe because it can only add findings, never suppress gate blockers. | Metric | Value | What It Means | |---|---|---| | label flip rate | 1.000 | The critic changes its verdict on every trial of identical input | | evidence drift rate | 1.000 | The critic invents different explanations every trial | | family migration rate | 0.000 | No seeded defect landed in an advisory family | | underclaim approvals | 0 | No defective plan got zero blockers | This is the finding I didn't expect: the critic is maximally non-deterministic, and it doesn't matter. The safety contract doesn't depend on the critic being consistent — it depends on the critic always finding something on defective plans. And it does. Takeaway:Deterministic gates own the under-claim direction preventing bad plans from slipping through , while code-enforced severity allowlists own the over-claim direction. The LLM critic can be 100% non-deterministic and still completely safe. v0.2.1 added the before/after numbers the community asked for: | Metric | Value | |---|---| | Latency approved p50 | 13.86s | | Latency escalated p50 | 27.82s | | Mean blockers per goal | 2.58 | | Mean advisories per goal | 1.86 | | Escalation decisions per 100 goals | 58.0 | | Mean LLM calls per goal | 1.4 | | Median revisions to resolution | 1.0 | The median revisions to resolution is 1.0 — most goals resolve in a single revision. The deterministic precondition closer and topological auto-repair are doing their job: they fix ordering and dependency defects without calling the LLM at all. | Metric | v0.2.0 | v0.2.1 | Delta | |---|---|---|---| | Goals swept | 170/170 | 170/170 | same | | Balanced approved | 73/73 100% | 73/73 100% | same | | Strict escalated | 97/97 100% | 96/97 99% | 1 transient provider error | | Adversarial aborted | 8/8 | 8/8 | same | | Verdict deltas vs prior | — | 30 | all attributable | | Deterministic tests | 90 | 1295 | +1205 contract tests for every new gate, evaluator, and schema | | Benchmarks | 3 | 3 | +operational, +boundary | | Code-review bugs fixed | 31 | 10 | — | | Field-test-found bugs | 0 | 0 | same | plan oscillation detected | 0 | 5 | 152 now fires | | Coverage | 91.62% | 91.58% | -0.04% accepted | 30 goals changed their verdict or reason code between v0.2.0 and v0.2.1. Every single one is attributable: converged stalled and revision cap reached depending on which specific blockers the critic found this time. plan oscillation detected in v0.2.1 0 in v0.2.0 . The 232 fix made the detector reachable under default config. It detects cycling earlier and terminates the loop sooner — saving LLM calls. mch-04-blast-radius hit planning unavailable — OpenRouter returned an error. The engine correctly failed closed escalated as error, not approved . Zero unexplained deltas. No delta is attributable to a code-review fix changing engine behavior — the fixes improved internal consistency gate ids, message quality, fault isolation without altering the approve/escalate decision logic. The arc across three releases tells a story about how field testing evolves: | Release | Goals | Cost | Issues Found by Field Test | Issues Found by Code Review | |---|---|---|---|---| | v0.1.0 | 157 | $0.30 | 10 | 0 | | v0.2.0 | 170 | $0.40 | 0 | 31 | | v0.2.1 | 170 | $0.49 | 0 | 10 | The field test went from a diagnostic found 10 issues in a greenfield engine to a validation tool confirmed 31 fixes in v0.2.0 to a regression gate confirmed 10 more fixes, diffed against the published baseline, zero unexplained deltas . The cost went from $0.30 to $0.49. The value went from finding bugs to proving their absence. In v0.1.0, only 1 of 10 issues was a traditional failure. The rest were design problems that would have shipped silently. The harness quietly lying about assertion results was worse than a crash would have been. Lesson: A field test harness that executes zero assertions for a module must fail hard — 0/0 is an error state, not a pass. Every issue that involved the LLM behaving unexpectedly was invisible in unit tests with hand-crafted inputs. The preconditions gate bug was hiding behind green tests. Lesson: Unit tests can't replace field tests. They test different things. v0.1.0 used the field test to find 10 issues ~$0.30 + 60 min . v0.2.0 used code review to find 31 bugs $0 + 2 hours . v0.2.1 found 10 more $0 + 1 hour . The field test found 0 in both cases. Lesson: For a mature engine with a proven corpus, code review before field test is the higher-leverage activity. The field test validates; the code review diagnoses. 30 verdict deltas between v0.2.0 and v0.2.1 are all attributable to gpt-4o-mini producing different findings across runs. The 218 live-critic boundary run measured this directly: label flip rate=1.0, evidence drift rate=1.0 — the critic changes its verdict and explanation on every trial of identical input. Yet family migration rate=0 and underclaim approvals=0 — it never under-claims a seeded defect. Lesson: Deterministic gates own the under-claim direction preventing bad plans from slipping through , while code-enforced severity allowlists own the over-claim direction. The LLM critic can be 100% non-deterministic and still completely safe. 3 goals now escalate with plan oscillation detected instead of revision cap reached . The signal detects cycling earlier and terminates the loop sooner. Lesson: The oscillation signal fires in practice 3/97 strict goals and saves LLM calls. 1 goal hit planning unavailable — OpenRouter returned an error. The engine correctly failed closed escalated as error, not approved . Lesson: Transient LLM provider errors are not engine defects; the engine fails closed. Latency p50 approved=13.86s , reviewer burden 2.58 blockers/goal , operator workload 58 decisions/100 goals . Median revisions to resolution = 1.0. Lesson: Operational baselines enable before/after comparison — the downstream-error-rate metric requires partner runner integration deferred to v0.3.0 . $0.49 for 170 goals + 60 boundary audits. Cheaper than a single developer-hour. There is no excuse not to field test your agent system. Lesson: The field test should run on every release. The cost is negligible. The deterministic gates run on every commit for free. The 1295 deterministic tests run in 4.7 seconds. The LLM field test runs on release for $0.49. Three tiers, each catching different things. Lesson: Unit tests can't replace the deterministic gate suite. The deterministic suite can't replace the field test. The field test can't replace code review. All three are needed. v0.3.0 will: approving authority through CLI/HTTP/MCP F-14 Article 4 of 5 in the PlannerCritic series. Series: Article 1: "I Ran 157 Agent Plans Against a Real LLM" https://dev.to/debashish ghosal/i-ran-157-agent-plans-against-a-real-llm-the-problem-wasnt-execution-it-was-planning-163j · Article 2: "I Told My LLM Critic to Be Adversarial" https://dev.to/debashish ghosal/i-told-my-llm-critic-to-be-adversarial-it-started-blocking-plans-for-being-not-thorough-enough-172 · Article 3: "The Planner Made the Same 3 Mistakes" https://dev.to/debashish ghosal/the-planner-made-the-same-3-mistakes-every-time-a-bigger-model-didnt-fix-it-3170 · Article 5: "I Tried to Prompt-Inject My Own Engine" Links: pip install planner-critic —