{"slug": "a-certification-that-changes-every-run-is-a-coin-flip-with-a-signature", "title": "A Certification That Changes Every Run Is a Coin Flip With a Signature", "summary": "A developer building the HivePlane agent control plane found that all three of their own previously built agents failed to import or run in the control-plane environment, and that LLM-dependent human-in-the-loop agents produced nondeterministic certification results. After replacing them with deterministic real agents wired through thin shims, the field test passed identically across three consecutive stack runs, with support-agent certifying 6/6 at p95 67 ms and eval-judge 4/4 at p95 126 ms under signed Ed25519 attestations.", "body_md": "The plan was clean, and I believed every word of it: three real agents I had already built — a repo guardian, a release-notes drafter, an incident commander — would register in [HivePlane](https://github.com/deghosal-2026/hiveplane), the control plane I'd just spent six weeks building, certify against their corpora, and prove the certified loop on *real* workloads.\n\nDay one, all three failed. None of the failures were the control plane's fault — which is exactly what made them interesting.\n\nIf you have ever tried to run an agent outside the repo it was born in, you already know where this is going.\n\n| Agent | What I expected | What actually happened | \n|---|---|---|\n| `release-narrator` | Import and certify | `from langgraph.checkpoint.sqlite import SqliteSaver` — the module isn't installed in the control-plane environment | \n| `ai-incident-commander` | Import and certify | Expects an installed `incident_commander` package; imports`incident_commander.ingest.input_dir` , which doesn't exist in the downloaded tree | \n| all three | Governed execution | Both import the external `openai` SDK — absent by design; HivePlane has its own provider seam, and agents cannot call providers directly | \n\nThese were **my own agents**, from my own repos, and they weren't drop-in runnable. That's a real finding about agent portability in general — \"works in my repo\" is not an interface — but it wasn't the finding the field test existed to produce.\n\nThe trio that *did* run made certification flaky. All three drive human-in-the-loop flows whose outputs depend on the LLM, and the early field-test runs recorded the failure mode plainly:\n\nthe real local model misclassified a bug-fix task as `changelog` and failed certification nondeterministically across runs.\n\nRead that again from the platform's point of view. A certification result that changes run to run isn't a certification. It's a coin flip with a signature on it. My thresholds were deterministic; my signal wasn't.\n\nI had to answer one question honestly: *is this field test supposed to measure the agents, or the control plane?*\n\nThe thesis of HivePlane is the loop — register, certify, gate admission, enforce budget and policy, pause and resume, deliver, audit. The agent is the workload; the plane is the product. So the Tier 1 subjects became **deterministic real agents** wired through thin shims:\n\n| Tier 1 subject | Adapter | What it exercises | \n|---|---|---|\n| `support-agent` (exectrace`agent-raw` ) | raw-worker | Read-first tool calls through the boundary, escalation to a destructive tool, 40 KB output truncation | \n| `eval-judge` (exectrace judge graph) | langgraph | StateGraph, cycles, human-review `interrupt()` , durable checkpointer resume | \n\nBoth are real agents — real code, real adapter dispatch, real policy and tool boundary — with a mock KB, mock tools, and a mock judge. Determinism is a *feature of the test design*: the same seed always produces the same category of run.\n\n**tip:** The model is still on the path where it matters. Identity binding, the provider seam, and the model-swap gate all run during certification and runs. What is mocked is the *judge's opinion*, not the plumbing.\n\nFrom the [field test report](https://github.com/deghosal-2026/hiveplane/blob/main/docs/field-test/v0.1.0/FIELD_TEST_REPORT.md):\n\n**S1 passed identically in three consecutive stack runs** — same tasks, same verdicts, same pass rates. Production certifications: support-agent 6/6 at p95 67 ms, eval-judge 4/4 at p95 126 ms, both at the 0.90 threshold with signed Ed25519 attestations.\n\nThat is what a benchmark gate needs to be before it can gate anything: **boring**. The signal now measures the control plane, not model drift.\n\nTwo findings only a live stack produces:\n\n`pagerduty.acknowledge` call — and `api.pagerduty.com` wasn't in the manifest's `sandbox.egress.allow`. The call was denied and the run died with the symptom `expected status='escalated', got None`. Unit tests mocked the host; only the live stack proved the allowlist was incomplete. Every host an agent's tools target must be allowed.`submitted.json`, `paused.json`, `approvals.json`, `resumed.json` written `POST /runs/{id}/resume` can return 409 when the approval's automatic re-dispatch already advanced the run — success is defined by the run reaching `completed`, which it did. The lesson, not the scenario, was the problem.\nThe restart scenario itself is worth naming: a paused LangGraph run survived a full `docker compose restart api` — event log intact, 8 events, re-attached from a durable checkpoint on the volume — then resumed to `completed`. The hard part worked on the first attempt, every time. It only ever *looked* stuck.\n\n**\"Drop-in runnable\" is an interface you have to design.** My own three agents failed in a new environment for three different reasons — missing dependency, missing package layout, wrong SDK. If your agent can't run outside its birth repo, it can't be operated by a fleet. That's a workload problem, not a platform problem.\n\n**Determinism is a certification requirement, not a convenience.** A benchmark that returns different verdicts for the same artifact can't be a gate — you can't build \"requires 'certified'\" on top of noise. Mock the judge, never the plumbing.\n\n**The field test's job is to catch what unit tests structurally cannot.** Egress allowlists, restart durability, approval flows — these only exist live. If your field test only re-proves your unit suite, you built an expensive unit test.\n\n**Write evidence before the boundary, not after the verdict.** Every scenario that \"hung\" was actually a scenario whose evidence hadn't been written yet. The same applies to any long-running verification job you own.\n\n`results/NOTES.md`` scripts/field-test.sh` regenerates everything from one run.\nNext in the series: the four deliberately bad agents I aimed at my own admission gate, and the exact refusal each one earned.\n\n**What's the most expensive \"it works on my machine\" failure you've hit with an agent — a missing dependency, a package layout, or something worse?**", "url": "https://wpnews.pro/news/a-certification-that-changes-every-run-is-a-coin-flip-with-a-signature", "canonical_source": "https://dev.to/debashish_ghosal/a-certification-that-changes-every-run-is-a-coin-flip-with-a-signature-bj9", "published_at": "2026-09-27 15:22:00+00:00", "updated_at": "2026-09-27 15:30:58.259506+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "mlops", "developer-tools"], "entities": ["HivePlane", "LangGraph", "OpenAI", "PagerDuty", "Ed25519"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/a-certification-that-changes-every-run-is-a-coin-flip-with-a-signature", "markdown": "https://wpnews.pro/news/a-certification-that-changes-every-run-is-a-coin-flip-with-a-signature.md", "text": "https://wpnews.pro/news/a-certification-that-changes-every-run-is-a-coin-flip-with-a-signature.txt", "jsonld": "https://wpnews.pro/news/a-certification-that-changes-every-run-is-a-coin-flip-with-a-signature.jsonld"}}