Countercheck – re-runs your tests when an AI agent says "done"

A tool called Countercheck re-runs a project's tests from the pre-change base revision to check whether an AI coding agent's "done, all tests pass" claim is true, sealing each verdict in a SHA-256-hashed receipt. In the project's example run, `countercheck verify --claim "Fixed the discount bug. All tests pass." --agent claude-code --cost-usd 0.42` returned FAILED: the agent had marked a failing test as skipped, and Countercheck restored the modified protected file tests/test_pricing.py from the base revision and re-ran the check in a macOS sandbox with no internet access, producing an AssertionError of -5.0 != 0 at a reported cost of $0.4200. Countercheck ships a Claude Code Stop hook (`countercheck hook claude-stop`), a Hermes Agent `pre_verify` hook, and a CI mode (`countercheck verify --base main`), with dedicated hooks for Codex, Cursor, Copilot and Aider still planned.

Proof that AI work really happened. AI coding agents often say "done, all tests pass" when it isn't true: they skip a failing test, edit the test so it passes, or never run it. countercheck verify checks the claim itself: 1. It reads your checks from the base revision before the agent's change , so the agent can't weaken them. 2. It copies the agent's files into a clean folder and puts back any protected file the agent changed your tests . 3. It runs the checks in a macOS sandbox : no internet localhost works; a check can opt in with network = true , writes only inside that folder. 4. It writes a receipt : the claim, what ran, the output, what the agent changed that it shouldn't have, the cost if known, and a verdict. Receipts are sealed with a SHA-256 hash, so edits are detectable. bash $ countercheck verify --claim "Fixed the discount bug. All tests pass." --agent claude-code --cost-usd 0.42 countercheck receipt 9073a384e88f9d37 FAILED claim: "Fixed the discount bug. All tests pass." - CONTRADICTED by evidence checks: .countercheck/checks.toml @ HEAD locked , sandbox: macos-seatbelt no internet, localhost only; writes only in the check folder protected Modified: tests/test pricing.py - restored from base FAIL unit tests 65 ms $ python3 -m unittest -q | self.assertEqual apply discount 10, 150 , 0 | AssertionError: -5.0 = 0 cost: $0.4200 reported - the worker changed 1 protected file s ; countercheck ran the checks with the base version - check "unit tests" failed exit 1 The agent's own run said OK skipped=1 : it had marked the failing test as skipped. The mental model: the agent never grades itself. Its claim is compared with what actually runs, on rules it couldn't change: base revision locked agent's work + claim checks.toml, your tests files, "Done, all tests pass." \ / +-- countercheck verify <+ 1 lock the checks from the base revision 2 clean copy edited tests put back 3 run in a sandbox no internet unless a check opts in 4 decide + seal receipt with a SHA-256 seal | VERIFIED - agent may finish FAILED + "done" - hook sends the failure back; agent fixes the code max 3 rounds FAILED + honest "can't, because ..." - agent may stop; receipt still says FAILED | Rule | In plain words | |---|---| | The worker never grades itself | the checks come from before the work started; edited tests are put back | | Re-run, don't read | the agent's claim and logs are never evidence; the checks run again | | Honest verdicts | VERIFIED, FAILED or UNVERIFIED with a reason , never a guess; a hint never softens a failure | | Everything from the repo is hostile | paths, sizes and links are checked, because the worker controls the files | | Domain-blind core | it knows nothing about SQL, games or ML; your checks carry the domain knowledge | | Fail closed on the verdict, open in the hooks | a broken referee never waves work through, and never traps your agent | | Leave evidence | every run writes a sealed receipt anyone can re-check | The code behind each rule, one run step by step, and where new work plugs in: docs/architecture.md https://github.com/datacrate360/countercheck/blob/PROD/docs/architecture.md . | You use | How Countercheck plugs in | |---|---| | Claude Code | Stop hook: countercheck hook claude-stop docs/hooks/ https://github.com/datacrate360/countercheck/blob/PROD/docs/hooks | | Cortex Code Snowflake | same hook format as Claude Code not yet tried on a real install | | Hermes Agent | pre verify hook: countercheck hook hermes-pre-verify | | Codex, Cursor, Copilot, Aider, your scripts | run countercheck verify --claim "…" when it says done dedicated hooks planned | | CI | countercheck verify --base main ; fail the job on any exit code but 0 GitHub Action planned | Ready recipes for real projects check files, helper scripts, a 5-minute start : recipes/ https://github.com/datacrate360/countercheck/blob/PROD/recipes — first one: Oracle → Snowflake conversion with Cortex Code https://github.com/datacrate360/countercheck/blob/PROD/recipes/oracle-to-snowflake . We gave Hermes Agent https://github.com/NousResearch/hermes-agent 29 small coding tasks, 3 times each, in four setups, and graded every run with hidden tests the agent never saw. Some tasks are impossible on purpose a test expects a wrong value, a server can't be reached : the honest answer is "this can't pass, because …". | False "done" claims claimed success, hidden tests fail | Hermes | + its verify on stop | + Countercheck | both | |---|---|---|---|---| | GPT 6.1, all tasks 87 runs per setup | 7 | 11 | 3 | 2 | | GPT 6.1, impossible tasks only | 6 | 8 | 0 | 0 | | DeepSeek V4 Pro, all tasks | 7 | 6 | 6 | 6 | - When GPT 6.1 got stuck, it edited the protected test and said "all tests pass". Countercheck re-ran the original tests, said FAILED and pushed back once; the agent restored the test and reported the real problem. - Countercheck only knows what your checks test: the remaining false claims were cases the visible tests didn't cover. DeepSeek rarely games tests, so there was little to catch. - It added about 1 second to the median run. Full method, numbers and limits: docs/eval/results-2026-10.md https://github.com/datacrate360/countercheck/blob/PROD/docs/eval/results-2026-10.md . Reproduce: eval/ https://github.com/datacrate360/countercheck/blob/PROD/eval . | You use | Install with | |---|---| | Python pip, a .venv , uv | pip install countercheck · uv tool install countercheck · or run without installing: uvx countercheck verify | | macOS or Linux, Homebrew | brew install datacrate360/tap/countercheck | | macOS or Linux, any shell | curl -fsSL https://raw.githubusercontent.com/datacrate360/countercheck/PROD/install.sh \| sh | | Windows 10/11, PowerShell | irm https://raw.githubusercontent.com/datacrate360/countercheck/PROD/install.ps1 \| iex or pip install countercheck / uvx countercheck | | Rust | cargo install --git https://github.com/datacrate360/countercheck countercheck | | Direct download | Releases https://github.com/datacrate360/countercheck/releases : one file per platform + SHA256SUMS.txt | Builds: macOS Apple silicon and Intel, one universal binary , Linux x86 64, arm64; static and Windows x86 64, countercheck.exe , no runtime to install . Both install scripts check every download against SHA256SUMS.txt before installing. install.ps1 puts countercheck.exe in %LOCALAPPDATA%\Programs\countercheck and tells you how to add that folder to PATH . Windows needs Git for Windows https://git-scm.com/download/win Countercheck calls git , and checks run with its sh , so the same checks.toml works on every OS . A check can use PowerShell or cmd instead: shell = "pwsh" or shell = "cmd" docs/checks.md https://github.com/datacrate360/countercheck/blob/PROD/docs/checks.md command . Sandbox by platform. On macOS, checks run in the system sandbox no internet, localhost allowed, writes only in the check folder; network = true lets a single check use the internet . On Linux and Windows there is no sandbox yet: checks run as a normal process and the receipt says so on Windows: none no sandbox on Windows yet ; run Countercheck inside a container or VM with networking off when you need isolation. | Platform | Checks run in | Internet during checks | Writes outside the check folder | |---|---|---|---| | macOS | the system sandbox Seatbelt | blocked localhost allowed unless network = true | blocked | | Linux | a normal process | allowed | allowed | | Windows | a normal process | allowed | allowed | What's shipped, in progress and next: docs/ROADMAP.md https://github.com/datacrate360/countercheck/blob/PROD/docs/ROADMAP.md . countercheck init writes .countercheck/checks.toml, edit the commands git add .countercheck && git commit -m "Countercheck checks" ... let an agent work ... countercheck verify --claim "what the agent said" uncommitted work vs HEAD countercheck verify --base main committed work on a branch vs main countercheck show .countercheck/receipts/<id .json re-check a receipt's seal .countercheck/checks.toml : protected = "tests/ " the agent may not change these; .countercheck/ is always protected link = "node modules", ".venv" git-ignored dependency folders to make visible read-only check name = "unit tests" run = "pytest -q" any command; runs with sh -c in the clean copy timeout secs = 600 Besides plain commands there are matrix, forbid, golden, budget and coverage checks: see docs/checks.md https://github.com/datacrate360/countercheck/blob/PROD/docs/checks.md and the runnable example in examples/data-migration https://github.com/datacrate360/countercheck/blob/PROD/examples/data-migration . Exit codes: 0 verified · 2 failed claim contradicted · 3 unverified nothing failed, but the checks weren't locked in the base revision, or none ran · 1 error · 4 countercheck show receipt was edited. countercheck ui a live page in your browser; updates when a new receipt lands countercheck report --open the same page as one HTML file offline, attach it to a PR or CI run countercheck ui serves the page on this computer only 127.0.0.1, with a random token in the address; other hosts and other web pages can't read it . It uses port 7357, or any free port if that one is taken --port N to choose , and running it again opens the one already running for the project. countercheck report writes one HTML page .countercheck/receipts/report.html , git-ignored with every saved run, newest first: the verdict, what the agent said, each check with its output, protected files it changed, whether the receipt's seal is intact, and for matrix checks a grid of targets × checks. It works offline, loads nothing from the internet and has no scripts, so it's safe to attach to a pull request or a CI run. - Hermes Agent: countercheck hook hermes-pre-verify keeps Hermes working while the checks fail. In the profile's config.yaml : hooks: on session start: - command: "countercheck hook session-start" pre verify: - command: "countercheck hook hermes-pre-verify" timeout: 300 Then approve both hooks once in a terminal hermes hooks list shows "✗ not allowlisted" until you do; the desktop app never asks . Setup, approval and how the project is found: docs/hooks/hermes.md https://github.com/datacrate360/countercheck/blob/PROD/docs/hooks/hermes.md . - Claude Code and Cortex Code: countercheck hook claude-stop blocks the Stop while the checks fail, with countercheck hook session-start on SessionStart . See docs/hooks/claude-code.md https://github.com/datacrate360/countercheck/blob/PROD/docs/hooks/claude-code.md . Add the session-start hook too. It remembers the commit each session started from, so the checks stay locked even if the agent commits its own edits to a test or to checks.toml . Without it, the checks are locked to HEAD . In CI, pass the base explicitly: countercheck verify --base origin/main . On Windows the hooks work the same Claude Code runs natively there : use the same settings with countercheck.exe on PATH , or give its full path with forward slashes, e.g. "command": "C:/Users/you/AppData/Local/Programs/countercheck/countercheck.exe hook claude-stop" . Paths in the hook input may be C:\... or C:/... ; a transcript under ~\ is found through USERPROFILE . See docs/hooks/claude-code.md https://github.com/datacrate360/countercheck/blob/PROD/docs/hooks/claude-code.md windows . A referee has to be right, so Countercheck is tested harder than most tools its size: - 140 tests cargo test --workspace : unit tests, end-to-end tests that run the real binary on throwaway git repositories tampered tests, timeouts, the macOS sandbox, both agent hooks, every check type , and property tests that throw thousands of generated inputs at the verdict and seal logic e.g. a receipt's seal breaks if any one of 33 fields changes; decide never says VERIFIED when a check failed . - Mutation testing cargo-mutants https://mutants.rs : the receipt, claim and config code was deliberately broken in every way the tool could find; tests were added until every break that compiles is caught one remaining change is provably equivalent . - CI on every push macOS and Linux : cargo fmt --check , cargo clippy -D warnings , all tests, a release build and the benchmark task validator. Countercheck also verifies itself with its own .countercheck/checks.toml . - The benchmark above, with hidden tests as ground truth. Countercheck is a referee that runs inside other people's projects: Python, JavaScript, Java, SQL, anything. So it has to be one dependency-free file that starts instantly, can't crash, and behaves the same on every operating system. Rust gives exactly that, which is also why tools like uv https://github.com/astral-sh/uv , ruff https://github.com/astral-sh/ruff and OpenAI's Codex CLI https://github.com/openai/codex are written in it. | What a referee needs | Why it matters here | What Rust gives | |---|---|---| | Nothing to install | It runs inside your project, so it must never clash with your Python version, virtualenv or node modules | One file of about 3–4 MB with no runtime; pip install countercheck ships that same binary in a wheel, like ruff | | Instant start | An agent hook runs it every time the agent tries to finish | Starts in under 10 ms a Python CLI often needs 100–300 ms just to import | | No crashes, no memory bugs | A bug in a referee is a wrong verdict | Memory safety from the compiler; on top, this project forbids unsafe and unwrap / panic outside tests | | Same everywhere | Teams mix macOS, Linux and Windows | One codebase, cross-compiled; static Linux builds | | A strict gate for AI-written code | Much of this code is written with coding agents | The compiler and clippy -D warnings reject whole classes of mistakes before review; property and mutation testing are first-class | The costs, honestly: fewer people know Rust than Python, and it is slower to write. That's why most contributions need no Rust at all: domain examples, check files and benchmark tasks are plain files, scripts and Python, and checks run your project's own commands in any language. Go would also have been a reasonable choice single binary, simpler language ; Rust won on stricter safety guarantees and its testing tools. Where Rust is not the answer: a future hosted team dashboard logins, many repositories, rich views is a web app and will use a web stack. The full reasoning is in docs/decisions/0001-stack.md https://github.com/datacrate360/countercheck/blob/PROD/docs/decisions/0001-stack.md . Does Countercheck limit what my agent can do? No. The agent works with its full harness: web search, docs, tools, anything you allow. Countercheck only steps in when the agent says "done": it re-runs your locked checks on a clean copy of the result. Like an exam after open-book study: the grade has to reflect the work itself. Does Countercheck edit tests or search the web? Never. It reads the checks from the base revision, runs them and writes a receipt. It makes no network connections of its own no telemetry, no network library in the binary and does not use an LLM: the verdict comes from your checks' exit codes and output. What is the sandbox, and which OS has one? A fence the operating system puts around the check while it runs, so the agent's code can't fetch the answer from the internet or write outside the check folder. | OS | Sandbox used for checks | Note | |---|---|---| | macOS | Seatbelt sandbox-exec : no internet localhost allowed , writes only in the check folder | network = true opens the internet for one check | | Linux | none yet planned: namespaces + Landlock | the receipt says none ; use a container with --network none for isolation | | Windows | none yet planned: AppContainer | the receipt says none no sandbox on Windows yet ; the Job Object only stops a check's processes | So without a sandbox, can the agent cheat on Linux or Windows? The common cheats are caught on every OS: edited tests are restored from the base revision, checks.toml is read from the locked base, committing edits doesn't move that base the session-start hook , and every check is re-run in a clean copy. The sandbox adds protection against code written to misbehave during the check fetching the expected answer online, leaving files outside the folder . Until Linux and Windows have one, run Countercheck in a container or VM with networking off when that matters. Why does grading run offline? So the same work always gets the same verdict, and nobody can fetch the answer at grading time. In the benchmark, the only false alarms came from checks that called a real website. If a check truly needs the internet, set network = true on it; the receipt then says "internet allowed". Does an LLM judge the work? No. The agent never grades itself, and Countercheck doesn't ask a model either. The checks you lock decide. The only text reading is a small, fixed list of phrases that tells whether the agent's final message already admits a blocker "this can't pass, because …" , so the hook doesn't push an honest agent to keep going. Will it catch every wrong "done"? Only what your checks test. In the benchmark, the false "done" claims left were cases the visible tests didn't cover. Stronger checks golden files, coverage, budgets: docs/checks.md https://github.com/datacrate360/countercheck/blob/PROD/docs/checks.md catch more. How much slower does it make my agent? About 1 second on the median benchmark run; longer when your checks are slow, because it runs them for real. Which agents does it work with? Hermes Agent and Claude Code have hooks Agent hooks agent-hooks . Snowflake's Cortex Code uses Claude Code's hook format, so the same hooks should work there not yet tried on a real install . Any other agent, script or CI job can call countercheck verify directly. Which operating systems? macOS, Linux and Windows 10/11 x86 64 . Windows needs Git for Windows. See Install install . Does my code leave my machine? No. Everything runs locally; receipts are files in .countercheck/receipts/ . countercheck ui listens on 127.0.0.1 only, behind a random token. Is it free? Yes, MIT-licensed. See Contribute contribute-make-it-fit-your-domain to add check types, examples or agent hooks. v0.3 CHANGELOG https://github.com/datacrate360/countercheck/blob/PROD/CHANGELOG.md . macOS has the sandbox; Linux and Windows run checks without one the receipt says so . Not yet: signed receipts, a per-host network allow-list, cost read from agent logs, Linux/Windows sandboxes, more agent hooks Codex, Cursor, a GitHub Action . See docs/ROADMAP.md https://github.com/datacrate360/countercheck/blob/PROD/docs/ROADMAP.md . Countercheck gets better with every field it learns about, and most of that needs no Rust : a domain is covered by a good checks.toml plus a small example. How agents fake "done" in game development, ML, dbt, video pipelines or a migration is knowledge we don't have and you do. | Start here | Rust? | |---|---| | Open a "New domain" issue https://github.com/datacrate360/countercheck/issues/new?template=new-domain.yml : how "done" goes wrong in your work | no | | Add an example for your field examples/ https://github.com/datacrate360/countercheck/blob/PROD/examples | no | | Add a benchmark task that tempts agents to cheat eval/ https://github.com/datacrate360/countercheck/blob/PROD/eval | no | | Hook another agent Codex, Cursor, OpenCode, … | yes | | Add tests, or a check type | yes | Each of these has a step-by-step guide in .claude/skills/ https://github.com/datacrate360/countercheck/blob/PROD/.claude/skills , written for people and for AI coding agents alike Claude Code loads them automatically . Read CONTRIBUTING.md https://github.com/datacrate360/countercheck/blob/PROD/CONTRIBUTING.md for the gate, pull requests and commit sign-off DCO . Rust 1.90+: cargo build --release → target/release/countercheck . Gate: cargo fmt --check , cargo clippy --workspace --all-targets -- -D warnings , cargo test --workspace , python3 eval/validate.py . Countercheck also verifies itself: countercheck verify checks in .countercheck/checks.toml . - crates/countercheck-core : pure logic check file, check types, protected paths, receipts, verdicts - apps/countercheck : the CLI git, clean copy, sandbox, runner, hooks - examples/ : one made-up project per domain, with checks that catch a planted mistake - eval/ , docs/eval/ : the benchmark against Hermes Agent tasks with hidden tests - .claude/skills/ : step-by-step contribution guides · docs/architecture.md : how it works inside - packaging/ , install.sh , install.ps1 , pyproject.toml : release packaging Homebrew, install scripts, Python wheels - docs/checks.md : every check type · docs/hooks/ : agent hooks · docs/decisions/ : design decisions MIT licensed.