Five times our coding agents said "it works", and it didn't

A small open-source lab in Switzerland that runs AI coding agents across roughly a dozen projects documented five cases where agents confidently reported success while being wrong, each traced to a measurement flaw rather than model error. The failures included running only tests/unit while CI runs tests/, stale .pyc bytecode surviving a size-neutral mutation restore, a pgrep pattern matching the agent's own command line, a test suite silently hitting a leftover server on the port, and an agent landing a branch tip instead of the reviewed SHA. The lab's conclusion is that an agent's claim is not evidence until a different agent re-runs the check on the exact commit.

I run a small open-source lab in Switzerland. Most of the code in our repositories is written by AI coding agents, from more than one vendor, working in parallel across about a dozen projects. They coordinate through a local hub we built for that purpose, and every change is supposed to be checked by a different agent before it lands. "Supposed to" is the interesting part. Over the past months our logs collected a set of moments where an agent reported success with complete confidence and was wrong. None of them was the model being stupid. Every one was a measurement problem: the agent checked something real, just not the thing it claimed to have checked. Here are five of them, and the rule each one left behind. An agent ran the test suite, got green, and reported the change as verified. CI went red a few minutes later. It had run tests/unit . CI runs tests/ . The integration tree was never measured. In the same session the same agent ran six of ten publication checks, the six it remembered, although the handover note it had just read said to run all of them. A subset produces a real, honest-looking result. Nothing in the output tells you it was a subset. Rule: enumerate the gate from its definition the CI config, the checklist file , never from memory. The report names the exact command that ran. During a mutation test we changed a source file, confirmed the test caught it, then restored the file from a backup. sha256 matched HEAD , git status was clean, inspect.getsource printed the correct code. The test still saw the mutated behaviour. The mutation happened to be size-neutral, and the restore landed in the same second as the cached .pyc . CPython checks a cached bytecode file against the source's modification time and size. Both matched, so it kept running stale bytecode from a byte-correct file. We proved it by unpacking the .pyc header. This can hand you a false PASS as easily as a false FAIL. Rule: a digest proves the file, not what is executing. For mutation testing, clear pycache or run with PYTHONDONTWRITEBYTECODE=1 . One agent reported for an entire session that its message watcher was armed. It never was. The status came from pgrep -f 'syn-wait. <name ' , which happily matched the agent's own command line, because the pattern string was in it. So the rest of the fleet treated that agent as reachable in real time, while it was only polling. Rule: a liveness claim needs evidence that excludes the checker itself: wait on a PID, a socket, or a marker file, never on a broad pattern match. An agent started a static server for end-to-end tests and stopped the old one by the PID it had recorded. The process actually listening on the port was a different one, left over from a clean baseline build. The new server failed to bind, the test runner quietly reused whatever answered on the port, and the suite came back green-ish against the wrong build. We only noticed because a known failure disappeared. Rule: after starting a server, read the PID from the port itself, check its working directory or the build id it serves, and record that with the run. A run where this was not checked does not count. An agent was told to land one specific, reviewed commit. It landed the branch tip instead, assuming the SHA it had been given was a copy-paste mistake. The two commits had byte-identical trees. They differed only in the author credit, and that credit mattered: in the landed one, the author and the reviewer were the same seat. We did not rewrite public history over it. We kept the commit and wrote a dated incident record naming the miss. CI can certify that code is correct; it cannot repair provenance. Rule: land the exact SHA that was reviewed, not whatever the branch points to. If the SHA looks wrong, ask before landing, not after. The common thread: an agent's claim is not evidence. "Tests pass" counts for nothing until a different agent has re-run them on the exact commit, with the exact command, and recorded both. That one rule has caught more of our mistakes than anything else. Tooling helps. Our hub keeps a durable event log, so every claim and handoff has a receipt you can go back to, and agents hold explicit claims on the files they touch so two of them never edit the same one silently. Reviews are done against a frozen object, not a moving branch. None of this makes agents smarter; it makes their mistakes visible early. Both pieces are open source: pip install synapse-channel If you run several agents in parallel: what is the failure that bit you hardest? I would genuinely like to compare notes in the comments. We are a pre-seed lab and this work is funded through GitHub Sponsors https://github.com/sponsors/anulum . This article was drafted with help from the same AI agents it describes; every incident above comes from our own logs, and I reviewed and edited the text.