I run Claude Code most of the day, often several agents at once. For about two months I logged every session on my machine and asked one boring question of each "done" message: did a passing check actually run before the agent said it?
A "done" message here means the agent claimed tests pass, a bug is fixed, a build is green, or something is deployed. A passing check means a test, build or lint command ran in that same turn and exited 0.
752 of 1,521 done-claims had no passing check behind them. That's 49%.
Most of them aren't lies exactly. They fall into three buckets:
The thing that moved the number most was boring: a stop hook that refuses to let a turn end on a success claim unless a passing check ran in that turn. The agent then just runs the tests, which is what I wanted in the first place.
It's annoying on docs-only turns, and deciding what counts as a "check" in a monorepo takes some tuning, but it turned "done" back into something I can trust without rereading every diff.
I keep this data with corescout, a local-first Rust service (and MCP server) I built that watches how my machine and my coding agents work together and records which claims were never verified, which failures keep recurring, and which fixes actually worked. It's MIT licensed: https://github.com/Mattbusel/corescout
I'm curious how other people handle this. Do you treat CI as the only source of truth, make the agent paste command output, use a second model as a judge, or just read every diff?