The plan was clean, and I believed every word of it: three real agents I had already built — a repo guardian, a release-notes drafter, an incident commander — would register in HivePlane, the control plane I'd just spent six weeks building, certify against their corpora, and prove the certified loop on real workloads.
Day one, all three failed. None of the failures were the control plane's fault — which is exactly what made them interesting.
If you have ever tried to run an agent outside the repo it was born in, you already know where this is going.
| Agent | What I expected | What actually happened |
|---|---|---|
release-narrator |
Import and certify | from langgraph.checkpoint.sqlite import SqliteSaver — the module isn't installed in the control-plane environment |
ai-incident-commander |
Import and certify | Expects an installed incident_commander package; importsincident_commander.ingest.input_dir , which doesn't exist in the downloaded tree |
| all three | Governed execution | Both import the external openai SDK — absent by design; HivePlane has its own provider seam, and agents cannot call providers directly |
These were my own agents, from my own repos, and they weren't drop-in runnable. That's a real finding about agent portability in general — "works in my repo" is not an interface — but it wasn't the finding the field test existed to produce.
The trio that did run made certification flaky. All three drive human-in-the-loop flows whose outputs depend on the LLM, and the early field-test runs recorded the failure mode plainly:
the real local model misclassified a bug-fix task as changelog and failed certification nondeterministically across runs.
Read that again from the platform's point of view. A certification result that changes run to run isn't a certification. It's a coin flip with a signature on it. My thresholds were deterministic; my signal wasn't.
I had to answer one question honestly: is this field test supposed to measure the agents, or the control plane?
The thesis of HivePlane is the loop — register, certify, gate admission, enforce budget and policy, and resume, deliver, audit. The agent is the workload; the plane is the product. So the Tier 1 subjects became deterministic real agents wired through thin shims:
| Tier 1 subject | Adapter | What it exercises |
|---|---|---|
support-agent (exectraceagent-raw ) |
raw-worker | Read-first tool calls through the boundary, escalation to a destructive tool, 40 KB output truncation |
eval-judge (exectrace judge graph) |
langgraph | StateGraph, cycles, human-review interrupt() , durable checkpointer resume |
Both are real agents — real code, real adapter dispatch, real policy and tool boundary — with a mock KB, mock tools, and a mock judge. Determinism is a feature of the test design: the same seed always produces the same category of run.
tip: The model is still on the path where it matters. Identity binding, the provider seam, and the model-swap gate all run during certification and runs. What is mocked is the judge's opinion, not the plumbing.
From the field test report: S1 passed identically in three consecutive stack runs — same tasks, same verdicts, same pass rates. Production certifications: support-agent 6/6 at p95 67 ms, eval-judge 4/4 at p95 126 ms, both at the 0.90 threshold with signed Ed25519 attestations.
That is what a benchmark gate needs to be before it can gate anything: boring. The signal now measures the control plane, not model drift.
Two findings only a live stack produces:
pagerduty.acknowledge call — and api.pagerduty.com wasn't in the manifest's sandbox.egress.allow. The call was denied and the run died with the symptom expected status='escalated', got None. Unit tests mocked the host; only the live stack proved the allowlist was incomplete. Every host an agent's tools target must be allowed.submitted.json, d.json, approvals.json, resumed.json written POST /runs/{id}/resume can return 409 when the approval's automatic re-dispatch already advanced the run — success is defined by the run reaching completed, which it did. The lesson, not the scenario, was the problem.
The restart scenario itself is worth naming: a d LangGraph run survived a full docker compose restart api — event log intact, 8 events, re-attached from a durable checkpoint on the volume — then resumed to completed. The hard part worked on the first attempt, every time. It only ever looked stuck.
"Drop-in runnable" is an interface you have to design. My own three agents failed in a new environment for three different reasons — missing dependency, missing package layout, wrong SDK. If your agent can't run outside its birth repo, it can't be operated by a fleet. That's a workload problem, not a platform problem.
Determinism is a certification requirement, not a convenience. A benchmark that returns different verdicts for the same artifact can't be a gate — you can't build "requires 'certified'" on top of noise. Mock the judge, never the plumbing.
The field test's job is to catch what unit tests structurally cannot. Egress allowlists, restart durability, approval flows — these only exist live. If your field test only re-proves your unit suite, you built an expensive unit test.
Write evidence before the boundary, not after the verdict. Every scenario that "hung" was actually a scenario whose evidence hadn't been written yet. The same applies to any long-running verification job you own.
results/NOTES.md`` scripts/field-test.sh regenerates everything from one run.
Next in the series: the four deliberately bad agents I aimed at my own admission gate, and the exact refusal each one earned.
What's the most expensive "it works on my machine" failure you've hit with an agent — a missing dependency, a package layout, or something worse?