I checked 101 "tests pass" claims from my AI coding agents. 35% weren't true.

A developer audited 101 "all tests pass" claims made by AI coding agents across two weeks of Claude Code and OpenAI Codex sessions and found 35 of them were not true at the moment they were made. Nearly all the false claims were stale rather than fabricated — the tests had genuinely passed earlier, but the agent kept editing code without re-running them. A prototype blocking gate that flags such claims reached only 73% accuracy (22 of 30 warnings correct) after five tuning rounds, below the developer's pre-set 90% bar, so the tool was shelved in favor of publishing the finding.

When an AI coding agent ends a session with "All tests pass ✅", I usually believe it. It's the last thing I read before I move on, commit, or hand the work to the next agent. So I went back and checked. Two weeks of my own sessions with Claude Code and OpenAI Codex, every success claim about tests, builds, lint or typecheck, compared against what had actually happened in the session up to that message. 35 of 101 claims were not true at the moment they were made. | | Claims | True | Stale | Failed | Delegated | |---|---|---|---|---|---| | Claude Code | 33 | 26 | 5 | 1 | 1 | | Codex | 68 | 37 | 29 | 0 | 2 | | Total | 101 | 63 | 34 | 1 | 3 | Almost none of the wrong claims were lies about a red test run. They were stale : the tests really had passed, but earlier. After that the agent kept editing code and never ran them again. In my data, the typical patterns were: By agent: 43 % of Codex claims and 18 % of Claude Code claims were not true. Don't read this as a model ranking. My Codex sessions were mostly long multi-agent runs, and the samples are small. Different work, not a fair comparison. The obvious next step was a hook that checks the claim before the agent may stop: "you edited src/x.ts after your last test run, run the tests again." I built it working name handfest and set myself a rule in advance: only ship a blocking gate if its warnings are right at least 90 % of the time. A gate that cries wolf gets turned off. It reached 73 % 22 of 30 warnings correct , after five rounds of tuning on the same data, so the real value is probably lower. The best single category, "stale", was right 84 % of the time. The remaining errors weren't parser bugs. They were questions of meaning: ssh … verify.log , read back later. Custom scripts with no recognizable name. So I dropped blocking and tried a soft version that only nudges the agent on "stale". Then I let it watch a fresh week of my normal work: 13 new claims, zero stale ones. In my day-to-day sessions the problem barely shows up. It piled up in the long multi-agent stretches. That's not enough to carry a tool, so I'm publishing the finding instead. None of this needs a tool: CLAUDE.md / AGENTS.md now say: | tail alone. tail , not of the tests, unless you use pipefail . It fooled my parser, and it can fool an agent too. If you run coding agents all day, try the count on your own logs: take the last ten "all green" messages and check whether anything was edited after the last check. I'd like to know whether you get close to one in three.