# A Governance-Catch Census: What Our Agent Harness Caught in One Day — and What Got Through First

> Source: <https://nxtg.ai/insights/governance-catch-census-2026-07-01>
> Published: 2026-07-27 00:00:00+00:00

# A Governance-Catch Census: What Our Agent Harness Caught in One Day — and What Got Through First

July 27, 2026 by NXTG.AI

**Publisher:** NXTG.AI · **Window:** 2026-07-01, a single operating day · **System:** a live production multi-agent system — long-running agent lanes across two machines doing real engineering and operations work.

Why publish this

Vendors publish what their monitoring *can* do. We wanted to know who publishes what their monitoring **actually caught, with the misses included**. In our **non-exhaustive surveyed set** — commercial vendors, safety-evaluation organisations, frontier labs and telemetry aggregators, probed on **2026-07-27** by a method that is disclosed internally but **not independently re-runnable, and which left four classes unprobed** — we found no party publishing an operational wrong-claim interception census with a public counting rule and public per-item evidence. **This is not evidence of universal absence.** **That set included us**, which is why this document exists.

The number here is not flattering, and that is the point: **at least 9 caught before they shipped, and at least one that reached a running server before it was caught.**

The premise behind such a census is not controversial. Independent research finds that AI models [choose the insecure way to write a task in 45% of cases](https://www.veracode.com/resources/analyst-reports/2025-genai-code-security-report/), and that experienced developers [misjudge their own AI-assisted output](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) — believing AI sped them up even when measurement showed it slowed them down. If AI output is wrong that often and self-assessment is that unreliable, the only honest question left is how much a harness *actually* catches, misses included. The figures below are our answer for one day — and, as §2 explains, they are deliberately reported as floors that do not support a rate.

1. The counting rule

Every item is assigned to exactly one of four classes. **The classes are reported separately and are never summed into a single "wrong claims" total** — blending them is the error that produced our earlier, retired figure.

| Class | Definition |
|---|---|
caught-wrong | A confirmed-wrong state claim caught before it shipped. Includes near-misses caught pre-ship. |
shipped-wrong | A wrong state that reached a running system before it was caught. Note the precision: not uncaught — caught, but only after reaching runtime. Counted against us. |
process-error | A real behavioural error that is not a falsifiable state claim (e.g. a runaway loop). Excluded from the state-claim numerator. |
verified-TRUE | A claim independently checked and found correct. A verification event, not an error — must never appear in a wrong-claims numerator. |

**Result for 2026-07-01** — *observed classified enumeration; both wrong-claim counts are floors, see §2:* caught-wrong **≥ 9** · shipped-wrong **≥ 1** · process-error = **1** · verified-TRUE = **1**

2. What these numbers do and do NOT support

**They do not support a catch rate.** An earlier draft of this document claimed the rate was "≤90%, a ceiling not a score."

**That inference is invalid and we are retracting it before publication rather than after.** caught-wrong ≥ 9 and shipped-wrong ≥ 1 are floors that move the quotient in

*opposite*directions — additional caught items raise it, additional shipped items lower it — so the pair constrains nothing about the true rate. The error was easy to miss precisely because it was dressed as conservatism.

**What is actually true:**

- The
**observed classified ratio is 9/10*** within this retrospective enumeration*. - The
**population catch rate is unidentified from these data.** There is no valid one-sided bound, in either direction. - Both counts are
**floors** over an enumeration known to be incomplete: unlogged catches**may** exist and are**unobservable to this ledger-derived enumeration**, and additional shipped-wrong is** unknown pending audit**.

**Selection is not preregistered.** This window was chosen because it had already been written up as an internal postmortem — it is **retrospective and outcome-salient**, not a randomly or prospectively selected day. That may bias the window in either direction, and no claim here should be read as a representative sample.

3. Per-item record — all 12 items, including the one that got through

| # | What was claimed / done | Class | How it was caught |
|---|---|---|---|
| 1 | "12 stuck message-queue rows should be expired" | caught-wrong | Probe at the authority source showed the rows were already-processed inbound messages, not orphans |
| 2 | "a scratch directory is not ignored — live bomb" | caught-wrong | An ignore-check against canonical origin settled it in one command; the alarm was a stale local view |
| 3 | A server fix was silently reverted (a shell metacharacter executed inside a payload) | shipped-wrong | Caught — but only AFTER the reverted state existed in the running server. A second layer of defence masked user impact. This is the one that got through. |
| 4 | An agent self-reported a commit as pushed | verified-TRUE | Fetch-then-judge confirmed it was pushed — a verification event, not an error |
| 5 | "the sync wedge may have eased" | caught-wrong | Lock-held ticks were bailing before commit — masked, not cleared |
| 6 | "the fix is to add the file to the path-pattern list" | caught-wrong (near-miss) | Wrong mechanism — it was the trigger set, not the pattern list; caught before it shipped |
| 7 | A watcher declared deploy-truth green | caught-wrong | It had matched a historical commit — no time filter on the probe |
| 8 | "the journal shows no device lines" (hardware fallback suspected) | caught-wrong | Faulty search pattern; a journal re-read refuted it |
| 9 | "cross-instance query is built" | caught-wrong | Roadmap, not built — self-caught mid-turn; the claim was factually false |
| 10 | "the cached-file removal fixed the dirty files" | caught-wrong | A skip-worktree bit had silently defeated it |
| 11 | Relay-loop inflation — every acknowledgement triggered a full cycle | process-error | Real and wrong, but a behaviour error, not a state claim — excluded from the numerator. Weakest row in this table: its only evidence is the retrospective postmortem plus an escalation record; no independent instrument survives and it is NOT independently certified. |
| 12 | "the two certs contradict each other on recall" | caught-wrong | The contradiction claim was itself false — an instrument-reconciliation error |

**Read row 3 first.** A census that lists only successes is marketing. Including the miss reduces one cherry-picking concern; it does **not** validate any other row. **Read row 11 second** — it is the row we are least able to evidence, and we are telling you so rather than letting it pad the table.

4. Evidence status — stated honestly, not claimed green

This table is a **redacted per-item summary**, not a set of independently checkable receipts. It provides no public evidence URL, content hash, or third-party attestation for any individual row. A reader can check our *counting rule* and our *reasoning*; they cannot yet independently verify any single row against source evidence.

**We are not claiming otherwise.** Publishing per-row evidence with stable public identifiers is the next build, and until it exists this document is a disclosed methodology plus a summary — which is more than we found in our non-exhaustive surveyed set, and less than proof.

5. Method, and how to challenge it

**Unit:** one state claim or action per row, assigned to exactly one class.**Independence:** the author of the original blended figure did**not** grade this partition. It was re-derived row-by-row from the operating record by a separate evidence pass, then adversarially reviewed by a**different vendor's model** on a separate context — which is what caught the invalid rate inference in §2 and the missing caveat on row 11, before publication.**The structural limit — a ledger-derived catch rate is ascertainment-biased.** You can only count the misses you eventually noticed. This does not mean such a rate*always*flatters in any given sample; it means the instrument**cannot distinguish perfect catching from undetected misses**. Because the ledger omits undetected misses by construction, a 100% observed ratio cannot distinguish perfect catching from unseen misses and is**not evidence that no misses occurred**— from any party, including any future figure from us.** What would change these numbers:**any additional shipped-wrong found in the pending audit. We will publish that too.** Other limits:**single-day window; single-organisation system; retrospective and outcome-salient selection; enumeration incomplete by construction; not a controlled experiment and no comparison arm. This is*practice observed*, not a benchmark.

6. What this is not

It is **not** a claim that our catch rate beats anyone's — we found no comparable published operational census **in our non-exhaustive surveyed set**, and this document does not establish a rate at all. Its value is that the counting rule is disclosed and the miss is included. **If you publish a competing census, we will link it.**

*Redaction note: internal host names, machine identifiers, absolute paths and internal repository identifiers are omitted; every row preserves the substance of the claim and the nature of the check that settled it.*