Who Verifies the Verifier?

A developer built a verification gate that forces Claude Code to prove each task in a real browser via the Kane CLI agent before it can stop, then audited the gate itself. Across twelve ORBITAL build tasks, eight failed at least once but only one was a real app defect; the rest were verifier problems such as slow agents, brittle assertions, and stale recordings, and six required human-reviewed overrides. The developer also found that their own write-up had misread the verifier's failures the same way the verifier misread the app, and that a standing self-override authorization weakened human oversight as the build progressed.

The premise of this series is that you should not take an agent's word that it is done. So I built a gate that makes Claude Code prove each task in a real browser before it may stop. The browser is driven by Kane CLI, which is itself an agent. The obvious question arrives late and uncomfortably: why would I take that agent's word either? This is the part of the story where the gate turns around and looks at itself. It has three layers: the verifier's failures, my own, and the rule that decides when to stop asking. ORBITAL is the densest build in the project, the one with the doubled chart from Part 1. Every task went through the gate, and the Stop hook wrote each attempt to an append-only activity log. Here is the whole build, from that log. 12 of 12 verified. One sweep catch. Six overrides. Eight tasks failed a check at least once. T1 failed on an assertion timing mismatch after the navigation it was checking had already happened, and passed on retry. Six more T2, T3, T4, T9, T10, T11 failed badly enough to need a human-reviewed override. For every one of the six, kane-cli 's own triage came back confirmed: false , family automation bug . The causes were specific: --author , all eight test files passed. The eighth was T7, the sweep catch from Part 1. That was the one real problem in the app across twelve tasks. Eight failures, one bug. Read by exit code, all eight look the same. Every raw failed result gets read against the verifier's structured verdict: verdict.confirmed , family , category , and confidence. Never the exit code. Then, before an override is allowed: kane-cli 's triage on the repeated failure. The same held across the other experiments. On the booking studio, six toast-timing failures were Kane's agent being too slow for a four second toast. On the todo build, a persistence check looked under the storage key the other build happened to use. A slow verifier, a brittle assertion, a stale recording. None of those is the app. All of them look exactly like the app from outside. This is the part I did not expect to write. While putting this series together I went back to the repo's build log, OBSERVATIONS-AND-REPORTINGS.md , and checked each ORBITAL claim against the raw activity log. My write-up had done to the verifier's failures exactly what the verifier had done to the app. What my write-up said against what the log said. The notes say the sweep "caught a real state-tracking defect" on the chart at T2. The log shows no sweep finding on T2 at all, only three scripted failures that triage called automation. The notes credit the sweep on T3 and T4; both were test problems. The notes call T9 "a genuine race"; the log shows all four export toasts confirmed in the same run and a redundant step failing afterwards. Kane's bug title fields made this easy to get wrong. Triage titles read like bug reports: "Replay misclassifies unlocked chart state". Lift that string out of its context, forget the confirmed: false next to it, and it reads as a finding. I lifted them. The first draft of Part 1 repeated them, before I checked. The repo's own evidence gif has the same problem. The README captions this one as a caught defect: A failed run on the chart lock, evidence pack 31fcd34b . Every failure on the chart task was triaged automation bug , so I now read this as the verifier failing, not the chart. Look at who cleared each override. T2 was recommended by a second Claude session and approved by me explicitly. T3 needed triage evidence and then my approval. T4, T9, T10 and T11 were cleared "under standing self-override authorization": a standing grant, not a fresh decision each time. The evidence supports each of those calls. But the process got weaker as the build went on, and the log shows it. A human in the loop who has pre-approved the loop is not really in it. All of which forces a design question I had to answer in code before I could answer it in prose. When the gate says no, how many times do you send the agent back? An agent told "try again" will try forever. It does not get tired or embarrassed. Somebody has to decide when the loop has stopped producing information and started producing noise. GuardianKane's answer is three. Every gate that can fail a task follows the same shape: increment attempts ; under three, set KANE FAILED and deny the stop with a reason; on the third, set BLOCKED NEEDS HUMAN . The details around the number matter more than the number: IN PROGRESS . CLAIMED DONE . The agent cannot re-assert done on the same code. The clearest log of the cap in action comes from a later project, an ecommerce demo built with the rebuilt GuardianKane and committed with its unedited history. The three-strike rule is unchanged there. Here is its first task, T0: | Time | Event | |---|---| | 20:15:30 | sweep FOUND ISSUE attempt 1/3 : no summary | | 20:27:58 | SECRET SCAN FAILED attempt 1/3 : server.js, high-entropy value assigned to "secret" | | 20:29:56 | sweep FOUND ISSUE attempt 2/3 : no summary | | 20:30:23 | sweep FOUND ISSUE attempt 3/3 : no summary | | 20:30:23 | BLOCKED NEEDS HUMAN after 3 sweep failures | | 22:11:02 | sweep found no issues, KANE VERIFIED | The escalation works. And the log taught me three things. The counter went backwards. The first sweep failure is attempt 1. Twelve minutes later the secret scan failure is also attempt 1. The hook only ever increments attempts ; nothing in it sets the field back to zero. But the counter lives in the task tracker, a YAML block in a markdown file the agent also edits, because the agent is the one who flips tasks to IN PROGRESS . Nothing guards that field. I do not have data on what reset it in this run. The log only shows that it was reset. That is the most important finding in this series. "Only the hook writes verification states" is a rule the agent is told, not one the file system enforces, and the same is true of the retry budget. A cap the capped party can edit is a suggestion. The fix is to keep attempts and the verification states in a file only the hook writes, and treat the tracker as a view. The last two sweeps were not a browser looking at a page. Attempt 3 started at 20:30:21 and reported an issue at 20:30:23. Two seconds, no summary. A sweep drives a real browser against a running app; a two-second verdict with nothing behind it is not evidence about the app. The escalation was reached mostly on verdicts like that, which is the lesson from the top of this post again. The human step is invisible. T0 sits blocked for about an hour and a half, then comes back clean. Whatever happened in between, a fix, a restart, a tracker edit, is not in the log, because the log only records what the hook does. For an audit trail, the most consequential step, a person deciding to reopen a blocked task, is the one it cannot see. I do not have data that three is the right number. I did not run the loop with two or five and compare. What I am more confident about is the structure around it: honest failures retry, tampering does not, the verifier's own failures do not count, and the budget has to live somewhere the agent cannot write. The hackathon version got the first three right. It does not weaken it. It sharpens it. The original claim was "do not trust the agent's self-report". The better claim is "do not trust any unverified claim, including the verifier's, including your own summary of the verifier". Each layer needs its structured evidence kept next to its conclusion, so the next reader can check the step instead of the sentence. In practice: read structured verdicts, not exit codes. Keep confidence and family next to every finding, everywhere it is quoted. And go back to the raw log when you write it up, because the summary is where the drift happens. The raw log was right the whole time. Everything that went wrong went wrong in a summary. That is also where the hackathon version runs out. A pass or fail line in a terminal cannot show you what the agent touched, which requirements are only designed and which are proven, or which failures were the checker. Answering that took a second build, and it is a different story. Part 3 of 3, and the last. Previously: the requirement nobody wrote https://abhinavpangaria.pages.dev/dispatches/the-requirement-nobody-wrote . The story continues in GuardianKane: it will not let your agent lie https://abhinavpangaria.pages.dev/dispatches/series/guardiankane , starting with the second build was mostly wiring https://abhinavpangaria.pages.dev/dispatches/the-second-build-was-mostly-wiring . Part 3 of Your Agent Might Actually Lie to You https://abhinavpangaria.pages.dev/dispatches/series/your-agent-might-lie . The original, with the full series navigation, is on my site https://abhinavpangaria.pages.dev/dispatches/who-verifies-the-verifier .