A follow-up to I built an AI dev harness that isn't allowed to trust itself.
The harness I wrote about isn't allowed to trust itself. That was the whole point, and it rested on one sentence:
nothing an agent produces closes without machine-checkable proof, and no irreversible action happens without a human.
I still believe it. But it contains an assumption I didn't notice I was making: that the dangerous thing is unchecked work.
Over one night I rebuilt both halves of that sentence on pi β an open, npm-distributed, forkable agent runtime β mostly to find out how much of my harness was discipline and how much was one host's shape. Two public repos came out of it: pi-eval for the proof half, and pi-gates for the irreversibility half.
Both halves failed first. They failed in the same shape, and it isn't the shape I'd been defending against. In each case the check ran, passed, and was proving something other than what it appeared to prove.
That's a correction to my own thesis, so here it is with the receipts.
The irreversibility half is a gate that denies git commit
and git push
unless I've typed an approval phrase at the keyboard that turn. Everything else an agent does in a repo is recoverable by reading a diff. A push isn't.
I wrote it, tested the pure logic, installed it, and asked my session to commit something without the phrase.
It refused. Then it told me:
the gate is functioning as intended β it's blocking the commit
No bash command ran. No tool_call
event fired. The gate was never invoked. The model had read about the rule earlier in the conversation and was enforcing it itself β and describing itself, accurately from its point of view, as the mechanism.
I nearly reported the gate verified off that transcript.
When I pushed back and said do not refuse, I need to see whether the gate blocks, it classified that as an attempted bypass and refused again.
The compounding detail is the part I'd want another engineer to take away. The model knew about the rule because my gate had told it. My first "armed" banner used ctx.ui.notify
, which lands in the conversation. The gate manufactured the cooperation that made it unverifiable.
Two fixes, both transferable:
Announce out of context. The banner moved to ctx.ui.setStatus
, which writes the TUI footer. I traced it: setStatus
β setExtensionStatus
in interactive-mode.js
, with no path into the model's context. A gate the model believes it owns is a gate it can be argued out of β and worse, one that sounds like a gate while being nothing of the kind.
Verify with no model in the loop. Gate correctness is now proven by firing synthetic events at the handler and reading the return:
const [r] = await pi.fire("tool_call",
{ toolName: "bash", input: { command: "git add -A && git commit -m x" } }, ctx);
assert.equal(r.block, true);
That file is the only artifact that proves the gate fires. When I finally ran the real thing in a clean session, the difference was unmistakable β the model ran the command and got denied:
$ cd /tmp/gitgate-test && git add note.txt && git commit -m "gate test 2"
BLOCKED β git commit/push needs an operator approval typed this turn.
Same outcome as the refusal. Completely different fact.
The proof half is a deterministic grader β fixed predicates, no LLM judge β so a score change means the output moved rather than the judge having a bad day.
Every task has to survive two machine checks before it enters a suite: a correct answer must PASS its predicate, and a plausible-but-wrong answer must FAIL it. That second one matters more than it looks, because the comparison is a paired sign test where ties are discarded. A predicate that can't fail a wrong answer turns a real difference into a tie. A lax grader doesn't mismeasure your suite β it blinds it.
The gate worked. Of 48 authored tasks it rejected 7, six of them for one cause: the number
grader takes the first number in a reply, and a worked solution's first number is an operand. Those predicates graded 3
against an expected 78
and failed the correct answer.
I was pleased with it. Then it admitted a task with expected = 52.34
.
48.45 Γ 1.08 = 52.326
, which rounds to 52.33.
Both models I tested answered 52.33 and were scored wrong. It's git-provable in both directions β commit 25d3466
adds "expected": 52.34
, and HEAD has 52.33
.
Here's why the gate couldn't catch it. I wrote the answer key. I also wrote the known-good answer used to verify the predicate. The same arithmetic slip was in both. Two independent-looking checks, one shared error, zero detection.
A gate verifies the predicate. It cannot verify the answer key.
The fix is a third gate: a separate agent derives every answer from the prompt alone, never shown my value, and disagreement kills the task. The predicate then runs against the independently derived answer β grading mine would be circular, since mine may have been reverse-engineered from the predicate.
Then the escalation, and it's the one that generalises furthest.
I compared thinking=medium
against thinking=off
across 100 tasks with three repetitions each. The result was clean: near-total ties, zero informative tasks, a cost ratio of essentially 1. Tidy, plausible, publishable.
--thinking
was inert. Pi's resolver takes the level from a model:level
pattern, not from the argument I was passing; it returned undefined
for every level, so the session fell back to the default. I had compared one config against itself.
"No measurable difference" was completely true and completely worthless.
What caught it was token counting I'd added an hour earlier for an unrelated reason: both sides reported ~116k reasoning tokens, and a thinking=off
run cannot do that.
Notice the shape. This check didn't pass for the wrong reason. It passed correctly, about the wrong question β and nothing downstream can detect that, because every artifact looks exactly like a successful measurement. So the fix isn't a better check, it's a refusal to start:
preflight: asked for thinking=off but the session resolved to medium.
Refusing to run β a run labelled with a config it did not use is worse than no run.
That is now my favourite line in either repo, and I'd put it in a category with the other refusals: a missing answer is an error and not a zero; a response truncated at the token cap is not an answer; two runs of different suites are not comparable and the tool says so rather than reporting the delta.
With the config actually applied, thinking=high
beat thinking=off
8β0, p=0.008.
Two of those wins weren't real.
thinking=off
answers in prose instead of thinking first. On one task it worked through the arithmetic, concluded "Friday" β correct β and an exact-match grader scored the whole reply. On another it listed matches at positions 0, 2, 4, 6, answered "4" β correct β and the extractor took the 0. The model was right both times.
Verbosity is not accuracy. But a config change that alters verbosity moves every position-sensitive grader in the same direction at once, so it doesn't look like noise. It looks like a finding.
I could have published 8β0 and explained in prose which two results I'd decided to disbelieve. That's the version where you have to trust me.
Instead the graders got a scope="last_line"
option, and re-scoring the same six runs produced 6β0, p=0.031 β selecting the same six tasks I'd picked by hand, mechanically. The verdict is weaker and I trust it more, because nobody has to take my word for which results were artifacts.
That's what "machine-checkable" is actually for. Not proving you're right. Removing your judgement from the places it can quietly do work.
And the answer it produces is a decision rather than a score, because cost sits next to it:
thinking=high won 6, lost 0, 84 tied, 10 unstable p = 0.031
$1.107 vs $0.445 2.49x cost
+6 tasks per 100 for two and a half times the money. Someone can act on that. A score alone isn't a decision.
The discipline has a price, and I'd rather show it than sell around it.
At 100 tasks the tool refused to rank two frontier models. Only 4 tasks separated them, and six is the floor at Ξ±=0.05 β no split of that data could have reached significance. So it says that, instead of reporting a tie:
Only 4 tasks separated them. Even a clean sweep of 4 could not clear p<0.05, so this suite cannot decide between them β that is a limit of the suite, not a finding about the configs.
"Cannot tell" and "they're the same" are different findings. Most tooling collapses them, and the collapse always favours having something to report.
There's a second rule underneath: a task where a config disagrees with itself carries no direction, and is discarded exactly like a tie. Necessary, because running one model three times against the same 100 tasks produces ~2.67 "informative" differences from within-model variance alone. Any real finding has to clear that.
Then I wrote in a README that this cost was fixable "with more repetitions." I tested that claim an hour later. It's backwards:
reps 4-6 9-1 unstable 13 informative 10 p=0.0215 decisive
reps 1-6 7-1 unstable 17 informative 8 p=0.0703 not
Twice the data, less power. Every extra repetition is another chance to observe a within-config disagreement, and the strict rule discards the task when it does. In the limit it throws away every genuinely stochastic task β precisely the ones carrying the most information about a noisy config. A conservative rule isn't a free choice, and the cost is measurable.
One more, because it's the part I'd most want checked if I were reading this. When the strict rule returned a non-significant result, I built a weaker rule that returned a significant one. The motivation was independent β I'd measured the discards-rise-with-reps property before checking whether a different rule moved any verdict β and that is also exactly what everyone who p-hacks believes about themselves.
So I wrote down the prediction, the refutation conditions and the fixed parameters, committed them before collecting fresh data, and committed the analysis script while the runs were still going. A pre-registration that leaves the analysis to be written afterwards only relocates the discretion.
One of three predictions failed β the conservative rule reached significance on fresh data when I'd predicted it wouldn't. I reported it as a failed prediction, because the whole point of writing it down first is that you don't get to reinterpret it after.
The open-runtime argument is usually ideological. Here it's concrete, and it's mostly about subtraction.
On Claude Code, my model gate needed three files and two hooks to answer one question: which model is running? A SessionStart hook wrote the id to .runtime_model_<session_id>
; the gate read it back; a second .runtime_model_ppid_<PPID>
file existed purely to cover the race where /clear
starts a new session before the startup hook writes the new file.
Pi hands the handler ctx.model
. All of it is gone β not refactored, deleted.
The git gate is a better version of the same story. The original kept its approval in a sentinel file, so it also needed a forge guard denying any command that so much as mentioned the sentinel's name, with documented residuals around assembled paths like P=.oae_approve; touch "${P}_pending"
. In pi the approval is a variable in the extension's closure, with no path from a bash command to it. That guard isn't hardened. It's unnecessary, and its residuals don't exist.
How much of a sophisticated system is design, and how much is scar tissue from its substrate? For these two gates: a lot of it was scar tissue.
But one thing didn't survive the move, and it's worth stating plainly rather than approximating. My harness had a Stop hook that refused to let a turn finish if source had changed without validation running. Pi has no blocking turn-end event β agent_end
, agent_settled
, turn_start
and turn_end
are all declared with no result type, so a handler literally cannot return a decision. If I need that rule I'll move the enforcement point to tool_call
before commit, and I'll call it a different gate rather than pretend it's a port.
And the trust root needed rewriting, in a way a careless port would miss. Pi's InputSource
is "interactive" | "rpc" | "extension"
β an extension can inject input. A gate that honoured every input event could be opened by the thing it exists to gate. The approval check is source === "interactive"
, and that single comparison is the whole security property.
I'm not retracting the original rule. The original's own logic is what forces the amendment.
The harness isn't allowed to trust itself. But I was trusting the part that does the not-trusting, and I had no mechanism that could tell me otherwise.
If an agent's work is unverified until a gate proves it, then:
A gate is also an agent's work.
It was written by someone under time pressure, it can pass for the wrong reason, and it can be verified by a subject who has read it and would like to be helpful. The check running is not evidence that the check fired. The commit not happening is not evidence that the gate blocked it.
Everything above is in two public repos, and every number in this post is reproducible from the run artifacts committed alongside them. That's deliberate, and it's the only part of my method I'd defend without qualification: if you can't point at the file, don't publish the number.
pi-eval Β· pi-gates Β· gradecore β the same grading engine behind a live drift board and a live crash test.