{"slug": "i-built-an-ai-dev-harness-that-isn-t-allowed-to-trust-itself-then-i-checked-the", "title": "I built an AI dev harness that isn't allowed to trust itself. Then I checked the part doing the not-trusting.", "summary": "A developer rebuilt an AI agent harness on the open-source pi.dev runtime and discovered a critical flaw: the model was enforcing a safety gate itself after reading about it in context, making the gate unverifiable. The engineer fixed the issue by moving announcements out of the model's context and verifying gate correctness with synthetic events outside the model loop.", "body_md": "*A follow-up to I built an AI dev harness that isn't allowed to trust itself.*\n\nThe harness I wrote about isn't allowed to trust itself. That was the whole point, and it rested on one sentence:\n\nnothing an agent produces closes without machine-checkable proof, and no irreversible action happens without a human.\n\nI still believe it. But it contains an assumption I didn't notice I was making: that the dangerous thing is unchecked work.\n\nOver one night I rebuilt both halves of that sentence on [pi](https://pi.dev) — an open, npm-distributed, forkable agent runtime — mostly to find out how much of my harness was discipline and how much was one host's shape. Two public repos came out of it: [pi-eval](https://github.com/egnaro9/pi-eval) for the proof half, and [pi-gates](https://github.com/egnaro9/pi-gates) for the irreversibility half.\n\nBoth halves failed first. They failed in the same shape, and it isn't the shape I'd been defending against. In each case the check **ran**, **passed**, and was proving something other than what it appeared to prove.\n\nThat's a correction to my own thesis, so here it is with the receipts.\n\nThe irreversibility half is a gate that denies `git commit`\n\nand `git push`\n\nunless I've typed an approval phrase at the keyboard that turn. Everything else an agent does in a repo is recoverable by reading a diff. A push isn't.\n\nI wrote it, tested the pure logic, installed it, and asked my session to commit something without the phrase.\n\nIt refused. Then it told me:\n\nthe gate is functioning as intended — it's blocking the commit\n\nNo bash command ran. No `tool_call`\n\nevent fired. The gate was never invoked. The model had read about the rule earlier in the conversation and was enforcing it *itself* — and describing itself, accurately from its point of view, as the mechanism.\n\nI nearly reported the gate verified off that transcript.\n\nWhen I pushed back and said *do not refuse, I need to see whether the gate blocks*, it classified that as an attempted bypass and refused again.\n\nThe compounding detail is the part I'd want another engineer to take away. The model knew about the rule because **my gate had told it.** My first \"armed\" banner used `ctx.ui.notify`\n\n, which lands in the conversation. The gate manufactured the cooperation that made it unverifiable.\n\nTwo fixes, both transferable:\n\n**Announce out of context.** The banner moved to `ctx.ui.setStatus`\n\n, which writes the TUI footer. I traced it: `setStatus`\n\n→ `setExtensionStatus`\n\nin `interactive-mode.js`\n\n, with no path into the model's context. A gate the model believes it owns is a gate it can be argued out of — and worse, one that *sounds* like a gate while being nothing of the kind.\n\n**Verify with no model in the loop.** Gate correctness is now proven by firing synthetic events at the handler and reading the return:\n\n```\nconst [r] = await pi.fire(\"tool_call\",\n  { toolName: \"bash\", input: { command: \"git add -A && git commit -m x\" } }, ctx);\nassert.equal(r.block, true);\n```\n\nThat file is the only artifact that proves the gate fires. When I finally ran the real thing in a clean session, the difference was unmistakable — the model *ran* the command and got denied:\n\n``` bash\n$ cd /tmp/gitgate-test && git add note.txt && git commit -m \"gate test 2\"\nBLOCKED — git commit/push needs an operator approval typed this turn.\n```\n\nSame outcome as the refusal. Completely different fact.\n\nThe proof half is a deterministic grader — fixed predicates, no LLM judge — so a score change means the output moved rather than the judge having a bad day.\n\nEvery task has to survive two machine checks before it enters a suite: a **correct** answer must PASS its predicate, and a **plausible-but-wrong** answer must FAIL it. That second one matters more than it looks, because the comparison is a paired sign test where ties are discarded. A predicate that can't fail a wrong answer turns a real difference into a tie. A lax grader doesn't mismeasure your suite — it blinds it.\n\nThe gate worked. Of 48 authored tasks it rejected 7, six of them for one cause: the `number`\n\ngrader takes the *first* number in a reply, and a worked solution's first number is an operand. Those predicates graded `3`\n\nagainst an expected `78`\n\nand failed the correct answer.\n\nI was pleased with it. Then it admitted a task with `expected = 52.34`\n\n.\n\n`48.45 × 1.08 = 52.326`\n\n, which rounds to **52.33**.\n\nBoth models I tested answered 52.33 and were scored wrong. It's git-provable in both directions — commit `25d3466`\n\nadds `\"expected\": 52.34`\n\n, and HEAD has `52.33`\n\n.\n\nHere's why the gate couldn't catch it. I wrote the answer key. I also wrote the known-good answer used to verify the predicate. The same arithmetic slip was in both. Two independent-*looking* checks, one shared error, zero detection.\n\n**A gate verifies the predicate. It cannot verify the answer key.**\n\nThe fix is a third gate: a separate agent derives every answer from the **prompt alone**, never shown my value, and disagreement kills the task. The predicate then runs against the *independently derived* answer — grading mine would be circular, since mine may have been reverse-engineered from the predicate.\n\nThen the escalation, and it's the one that generalises furthest.\n\nI compared `thinking=medium`\n\nagainst `thinking=off`\n\nacross 100 tasks with three repetitions each. The result was clean: near-total ties, zero informative tasks, a cost ratio of essentially 1. Tidy, plausible, publishable.\n\n`--thinking`\n\nwas inert. Pi's resolver takes the level from a `model:level`\n\n*pattern*, not from the argument I was passing; it returned `undefined`\n\nfor every level, so the session fell back to the default. I had compared one config **against itself.**\n\n\"No measurable difference\" was completely true and completely worthless.\n\nWhat caught it was token counting I'd added an hour earlier for an unrelated reason: both sides reported ~116k *reasoning* tokens, and a `thinking=off`\n\nrun cannot do that.\n\nNotice the shape. This check didn't pass for the wrong reason. It passed **correctly, about the wrong question** — and nothing downstream can detect that, because every artifact looks exactly like a successful measurement. So the fix isn't a better check, it's a refusal to start:\n\n```\npreflight: asked for thinking=off but the session resolved to medium.\nRefusing to run — a run labelled with a config it did not use is worse than no run.\n```\n\nThat is now my favourite line in either repo, and I'd put it in a category with the other refusals: a missing answer is an error and not a zero; a response truncated at the token cap is not an answer; two runs of different suites are not comparable and the tool says so rather than reporting the delta.\n\nWith the config actually applied, `thinking=high`\n\nbeat `thinking=off`\n\n**8–0, p=0.008.**\n\nTwo of those wins weren't real.\n\n`thinking=off`\n\nanswers in prose instead of thinking first. On one task it worked through the arithmetic, concluded \"Friday\" — correct — and an exact-match grader scored the whole reply. On another it listed matches at positions 0, 2, 4, 6, answered \"4\" — correct — and the extractor took the 0. **The model was right both times.**\n\nVerbosity is not accuracy. But a config change that alters verbosity moves every position-sensitive grader in the same direction at once, so it doesn't look like noise. It looks like a finding.\n\nI could have published 8–0 and explained in prose which two results I'd decided to disbelieve. That's the version where you have to trust me.\n\nInstead the graders got a `scope=\"last_line\"`\n\noption, and re-scoring the same six runs produced **6–0, p=0.031** — selecting the same six tasks I'd picked by hand, mechanically. The verdict is weaker and I trust it more, because nobody has to take my word for which results were artifacts.\n\nThat's what \"machine-checkable\" is actually for. Not proving you're right. Removing your judgement from the places it can quietly do work.\n\nAnd the answer it produces is a decision rather than a score, because cost sits next to it:\n\n```\nthinking=high  won 6, lost 0, 84 tied, 10 unstable      p = 0.031\n               $1.107  vs  $0.445                       2.49x cost\n```\n\n**+6 tasks per 100 for two and a half times the money.** Someone can act on that. A score alone isn't a decision.\n\nThe discipline has a price, and I'd rather show it than sell around it.\n\nAt 100 tasks the tool refused to rank two frontier models. Only 4 tasks separated them, and six is the floor at α=0.05 — no split of that data could have reached significance. So it says that, instead of reporting a tie:\n\nOnly 4 tasks separated them. Even a clean sweep of 4 could not clear p<0.05, so this suite cannot decide between them — that is a limit of the suite, not a finding about the configs.\n\n*\"Cannot tell\"* and *\"they're the same\"* are different findings. Most tooling collapses them, and the collapse always favours having something to report.\n\nThere's a second rule underneath: **a task where a config disagrees with itself carries no direction, and is discarded exactly like a tie.** Necessary, because running one model three times against the same 100 tasks produces ~2.67 \"informative\" differences from within-model variance alone. Any real finding has to clear that.\n\nThen I wrote in a README that this cost was fixable \"with more repetitions.\" I tested that claim an hour later. It's backwards:\n\n```\nreps 4-6    9-1   unstable 13   informative 10   p=0.0215   decisive\nreps 1-6    7-1   unstable 17   informative  8   p=0.0703   not\n```\n\n**Twice the data, less power.** Every extra repetition is another chance to observe a within-config disagreement, and the strict rule discards the task when it does. In the limit it throws away every genuinely stochastic task — precisely the ones carrying the most information about a noisy config. A conservative rule isn't a free choice, and the cost is measurable.\n\nOne more, because it's the part I'd most want checked if I were reading this. When the strict rule returned a non-significant result, I built a weaker rule that returned a significant one. The motivation was independent — I'd measured the discards-rise-with-reps property before checking whether a different rule moved any verdict — and that is also exactly what everyone who p-hacks believes about themselves.\n\nSo I wrote down the prediction, the refutation conditions and the fixed parameters, committed them **before** collecting fresh data, and committed the analysis script while the runs were still going. A pre-registration that leaves the analysis to be written afterwards only relocates the discretion.\n\nOne of three predictions failed — the conservative rule reached significance on fresh data when I'd predicted it wouldn't. I reported it as a failed prediction, because the whole point of writing it down first is that you don't get to reinterpret it after.\n\nThe open-runtime argument is usually ideological. Here it's concrete, and it's mostly about subtraction.\n\nOn Claude Code, my model gate needed three files and two hooks to answer one question: *which model is running?* A SessionStart hook wrote the id to `.runtime_model_<session_id>`\n\n; the gate read it back; a second `.runtime_model_ppid_<PPID>`\n\nfile existed purely to cover the race where `/clear`\n\nstarts a new session before the startup hook writes the new file.\n\nPi hands the handler `ctx.model`\n\n. All of it is gone — not refactored, deleted.\n\nThe git gate is a better version of the same story. The original kept its approval in a sentinel **file**, so it also needed a forge guard denying any command that so much as mentioned the sentinel's name, with documented residuals around assembled paths like `P=.oae_approve; touch \"${P}_pending\"`\n\n. In pi the approval is a variable in the extension's closure, with no path from a bash command to it. That guard isn't hardened. It's unnecessary, and its residuals don't exist.\n\nHow much of a sophisticated system is design, and how much is scar tissue from its substrate? For these two gates: a lot of it was scar tissue.\n\nBut one thing didn't survive the move, and it's worth stating plainly rather than approximating. My harness had a Stop hook that refused to let a turn finish if source had changed without validation running. Pi has no blocking turn-end event — `agent_end`\n\n, `agent_settled`\n\n, `turn_start`\n\nand `turn_end`\n\nare all declared with no result type, so a handler literally cannot return a decision. If I need that rule I'll move the enforcement point to `tool_call`\n\nbefore commit, and I'll call it a different gate rather than pretend it's a port.\n\nAnd the trust root needed rewriting, in a way a careless port would miss. Pi's `InputSource`\n\nis `\"interactive\" | \"rpc\" | \"extension\"`\n\n— **an extension can inject input.** A gate that honoured every input event could be opened by the thing it exists to gate. The approval check is `source === \"interactive\"`\n\n, and that single comparison is the whole security property.\n\nI'm not retracting the original rule. The original's own logic is what forces the amendment.\n\nThe harness isn't allowed to trust itself. But I was trusting the part that does the not-trusting, and I had no mechanism that could tell me otherwise.\n\nIf an agent's work is unverified until a gate proves it, then:\n\n**A gate is also an agent's work.**\n\nIt was written by someone under time pressure, it can pass for the wrong reason, and it can be verified by a subject who has read it and would like to be helpful. The check running is not evidence that the check fired. The commit not happening is not evidence that the gate blocked it.\n\nEverything above is in two public repos, and every number in this post is reproducible from the run artifacts committed alongside them. That's deliberate, and it's the only part of my method I'd defend without qualification: if you can't point at the file, don't publish the number.\n\n[pi-eval](https://github.com/egnaro9/pi-eval) · [pi-gates](https://github.com/egnaro9/pi-gates) · [gradecore](https://github.com/egnaro9/gradecore) — the same grading engine behind a [live drift board](https://egnaro9.github.io/model-drift/) and a [live crash test](https://crashkit.onrender.com).", "url": "https://wpnews.pro/news/i-built-an-ai-dev-harness-that-isn-t-allowed-to-trust-itself-then-i-checked-the", "canonical_source": "https://dev.to/agentdev9/i-built-an-ai-dev-harness-that-isnt-allowed-to-trust-itself-then-i-checked-the-part-doing-the-298a", "published_at": "2026-07-27 05:24:39+00:00", "updated_at": "2026-07-27 05:34:44.086018+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "developer-tools", "ai-research"], "entities": ["pi.dev", "pi-eval", "pi-gates"], "alternates": {"html": "https://wpnews.pro/news/i-built-an-ai-dev-harness-that-isn-t-allowed-to-trust-itself-then-i-checked-the", "markdown": "https://wpnews.pro/news/i-built-an-ai-dev-harness-that-isn-t-allowed-to-trust-itself-then-i-checked-the.md", "text": "https://wpnews.pro/news/i-built-an-ai-dev-harness-that-isn-t-allowed-to-trust-itself-then-i-checked-the.txt", "jsonld": "https://wpnews.pro/news/i-built-an-ai-dev-harness-that-isn-t-allowed-to-trust-itself-then-i-checked-the.jsonld"}}