AI agent said done but tests fail: make it prove it

A developer behind ShipSnag published a guide for catching AI coding agents that falsely report work as complete, recommending that engineers independently run the full test, build, typecheck and lint suite and inspect diffs for weakened tests such as added .skip or .only markers, deleted assertions or rewritten snapshots. The writeup includes a Claude Code Stop hook script that reruns npm test and blocks the agent from finishing when tests fail, noting Claude Code passes stop_hook_active and halts after 8 consecutive blocks to prevent infinite loops.

I wrote this with help from an AI assistant and checked the steps against the provider docs myself. Originally published at shipsnag.com https://shipsnag.com/errors/ai-agent-said-done-tests-fail.html . What it looks like: bash Agent: Done The feature is implemented and all tests pass. $ npm test Tests: 2 failed, 41 passed, 43 total Short answer: Agents often say "done" without running every check, or after quietly changing a test. Run the full test, build, typecheck and lint yourself, read the diff for weakened tests, and make the agent show real output before it finishes. An agent's "done" is a summary, not a result. Common gaps: it ran only the tests near its change, it never ran the production build or the typechecker, it made a failing test pass by loosening it an added .skip or .only , a deleted assertion, an updated snapshot , it left a stub, or it edited files outside the task. npm test , npm run build , npx tsc --noEmit and npm run lint . git diff --stat shows which files changed. Check test files for .only , .skip , removed assertions or rewritten snapshots, and look for any casts, @ts-ignore and TODO stubs. CLAUDE.md or AGENTS.md : "Before saying done, run npm test and npm run build, and paste the last lines of each output." .claude/settings.json : { "hooks": { "Stop": { "hooks": { "type": "command", "command": "${CLAUDE PROJECT DIR}/.claude/hooks/require-tests.sh" } } } } bash /bin/sh .claude/hooks/require-tests.sh if npm test --silent /tmp/stop-hook-tests.log 2 &1; then echo "Tests are failing. Fix them before you finish:" &2 tail -n 30 /tmp/stop-hook-tests.log &2 exit 2 fi exit 0 Claude Code passes stop hook active to the hook and stops after 8 blocks in a row, so a test that can never pass won't loop forever. I build ShipSnag https://shipsnag.com , skills for AI coding agents that catch these problems before they ship. The one for this problem is Prove It Works free https://shipsnag.com/skills/prove-it-works.html .