Agent's tests passed. Prove they could have failed

Your agent's tests passed. Prove they could have failed.

mustfail reverts the source files your coding agent just changed, keeps the tests it just wrote exactly as written, and runs the checks again. A check that still passes did not test the work.

Zero runtime dependencies. Python 3.8 or newer, and git.

mustfail 0.1.0    HEAD d28ac3e (dirty)    baseline d28ac3e
  auto: working tree is dirty, baseline is HEAD
3 files changed: 1 source, 2 test, 0 exempt

CHECKS  (positive control: the work as written, clean worktree)
  unit         pass  exit 0      0.0s

NEGATIVE CONTROL  (1 source file(s) reverted to d28ac3e, tests kept as written)
  unit         NOT DETECTED           exit 0
       passed with 1 source file(s) reverted to d28ac3e

VERDICT  fail (exit 3)
  every check passed, and unit still passed with the work reverted. Green is not evidence here.

An agent rarely forges a receipt. The receipt is usually real: the suite ran, it was green, the summary was accurate. What breaks is the assay underneath it. The agent widens a mock until the unit under test never executes, asserts toBeDefined() instead of a value, writes a test against the stub it wrote five minutes earlier, or pipes an exit code through tee so the status is swallowed. Then it runs the suite, gets a genuine green, and tells you it is done.

Tools that check whether the suite ran cannot see any of that. Tools that diff for tampering catch deletion, not substitution, and substitution looks like work.

There is one question that separates a test from a formality: if this change vanished, would anything have noticed?

pipx install mustfail          # or: pip install mustfail

Or run it from a clone with no install at all:

git clone https://github.com/greenlitbooks/mustfail
PYTHONPATH=mustfail/src python3 -m mustfail --help
mustfail init        # writes a starter mustfail.json
mustfail doctor      # is the gate wired, and would it fire?
mustfail run         # positive control, negative control, verdict
mustfail scan        # static scan only, milliseconds, no execution
mustfail selftest    # run the corpus and check it against its manifest

Minimal config:

{
  "version": 1,
  "source": ["src/**"],
  "checks": [
    { "id": "unit", "run": "npm test", "must_exit": 0, "control": "revert-source" }
  ]
}

Exit codes: 0 fine, 1 internal error, 2 a declared check failed, 3 checks passed but a control was not detected, 4 a blocking swallow finding, 5 invalid config.

1. Isolate. A throwawaygit worktree is created at the baseline commit and overwritten with your work. Your own working tree is never touched. A tool whose job is reverting files has exactly one unacceptable failure mode, and this removes it.
2. Positive control. Every check runs against the work as written. All must pass. Without this step, a worktree missing its dependencies would "detect" everything and the tool would be confidently useless.
3. Negative control. Every changed source file is reverted to the baseline. Every test file stays exactly as the agent wrote it. The checks run again, and each one is expected to fail.
4. Classify. A failure caused only by a missing import is reported as a weak signal and not counted, because a new test importing a new function always breaks on revert. That proves the symbol is referenced, not that anything is asserted.
5. Localize. Only when a control was not detected, and only then, each source file is reverted alone to name the file nothing covers.
6. Scan. Independently and in milliseconds, the check commands and your CI config are scanned for constructs that discard a failing exit code.

A check is detecting when its control failed in a way that counts. By default one detecting check is enough, because the question is whether anything would have noticed, not whether everything would have. A type checker or a corpus runner will not be falsified by every source change, and failing the gate for that is how gates get uninstalled.

Set require.control_detects to "all" when every check really is meant to cover every source file. Set on_weak_signal to "fail" on a check where a load-time break should not count.

Checks that did not detect are still reported, with the specific files named, even when the run passes. That reporting is where most of the value is: it is how you find out which part of a change nothing tests.

This costs roughly twice your suite. If your tests take ten minutes, do not put it on Stop. Put it on pre-push and in CI.

mustfail hook    # prints the block to paste into .claude/settings.json

The Stop hook blocks the turn with a specific reason when a control was not detected, then stands down after repeating itself so it can never trap a session. See docs/HOOKS.md for the full contract and the failure modes that silently disable a gate.

Nothing about the engine is Claude-specific. mustfail never reads an agent transcript. Everything it knows comes from git, the filesystem, and exit codes, which is why the same gate works under Codex, Cursor, a pre-push hook, plain CI, or a human being.

corpus/ holds hand-built repositories, each with one documented mechanism for manufacturing a green run, each with the verdict it must produce recorded in MANIFEST.json. mustfail selftest builds every one in a throwaway git repo and checks the result.

pass 001-test-written-against-the-stub
pass 002-assertion-free-test
pass 003-mock-swallows-the-unit
pass 004-pipe-swallows-exit-code
pass 005-honest-fix-and-test
pass 006-docs-only-change
pass 007-continue-on-error

7/7 fixtures behaved as the manifest says.

005-honest-fix-and-test is the one that matters most. It is a real fix with a real test, and it must stay green. A gate that fails good work gets uninstalled in a week, so the false-positive rate is the quality metric here, not the detection rate. There are no detection percentages in this README and there will not be any until real runs have been measured and the raw data published next to the number.

Plenty. docs/LIMITS.md is not a formality: reverting whole files is crude, a pure refactor will look undetected, the failure classification is string matching, the static scan is regex rather than a parser, and an agent that can edit files can edit the config. Read it before you rely on this.

This is a small addition to a crowded shelf. Mutation testing has done the hard version of this idea for decades, and at least a dozen agent-gating tools solve the neighbouring problem. docs/PRIOR-ART.md names and links all of them, including the ones you should probably use instead.

Issues and pull requests welcome. mustfail gates its own repository, so a change that nothing tests will be blocked by the tool itself.

MIT. Built and maintained by Greenlit Books, whose field guide on how AI fakes "Done" is free and needs no email address.