AI agent said done but tests fail: make it prove it

I wrote this with help from an AI assistant and checked the steps against the provider docs myself. Originally published at shipsnag.com.

What it looks like:

Agent: Done! The feature is implemented and all tests pass.

$ npm test
Tests:       2 failed, 41 passed, 43 total

Short answer:

Agents often say "done" without running every check, or after quietly changing a test. Run the full test, build, typecheck and lint yourself, read the diff for weakened tests, and make the agent show real output before it finishes.

An agent's "done" is a summary, not a result. Common gaps: it ran only the tests near its change, it never ran the production build or the typechecker, it made a failing test pass by loosening it (an added .skip or .only, a deleted assertion, an updated snapshot), it left a stub, or it edited files outside the task.

npm test, npm run build, npx tsc --noEmit and npm run lint. git diff --stat shows which files changed. Check test files for .only, .skip, removed assertions or rewritten snapshots, and look for any casts, @ts-ignore and TODO stubs.CLAUDE.md or AGENTS.md: "Before saying done, run npm test and npm run build, and paste the last lines of each output.".claude/settings.json:

   {
     "hooks": {
       "Stop": [
         { "hooks": [ { "type": "command",
           "command": "${CLAUDE_PROJECT_DIR}/.claude/hooks/require-tests.sh" } ] }
       ]
     }
   }
bash
   #!/bin/sh
   if ! npm test --silent >/tmp/stop-hook-tests.log 2>&1; then
     echo "Tests are failing. Fix them before you finish:" >&2
     tail -n 30 /tmp/stop-hook-tests.log >&2
     exit 2
   fi
   exit 0

Claude Code passes stop_hook_active to the hook and stops after 8 blocks in a row, so a test that can never pass won't loop forever.

I build ShipSnag, skills for AI coding agents that catch these problems before they ship. The one for this problem is Prove It Works (free).