{"slug": "countercheck-re-runs-your-tests-when-an-ai-agent-says-done", "title": "Countercheck – re-runs your tests when an AI agent says \"done\"", "summary": "A tool called Countercheck re-runs a project's tests from the pre-change base revision to check whether an AI coding agent's \"done, all tests pass\" claim is true, sealing each verdict in a SHA-256-hashed receipt. In the project's example run, `countercheck verify --claim \"Fixed the discount bug. All tests pass.\" --agent claude-code --cost-usd 0.42` returned FAILED: the agent had marked a failing test as skipped, and Countercheck restored the modified protected file tests/test_pricing.py from the base revision and re-ran the check in a macOS sandbox with no internet access, producing an AssertionError of -5.0 != 0 at a reported cost of $0.4200. Countercheck ships a Claude Code Stop hook (`countercheck hook claude-stop`), a Hermes Agent `pre_verify` hook, and a CI mode (`countercheck verify --base main`), with dedicated hooks for Codex, Cursor, Copilot and Aider still planned.", "body_md": "**Proof that AI work really happened.**\n\nAI coding agents often say \"done, all tests pass\" when it isn't true: they skip a failing test, edit the test so it\npasses, or never run it. `countercheck verify` checks the claim itself:\n\n1. It reads your checks from the **base revision** (before the agent's change), so the agent can't weaken them.\n2. It copies the agent's files into a **clean folder** and puts back any protected file the agent changed (your tests).\n3. It runs the checks in a **macOS sandbox** : no internet (localhost works; a check can opt in with`network = true` ),\nwrites only inside that folder.\n4. It writes a **receipt** : the claim, what ran, the output, what the agent changed that it shouldn't have, the cost if\nknown, and a verdict. Receipts are sealed with a SHA-256 hash, so edits are detectable.\n\n``` bash\n$ countercheck verify --claim \"Fixed the discount bug. All tests pass.\" --agent claude-code --cost-usd 0.42\ncountercheck receipt 9073a384e88f9d37  FAILED\n  claim:    \"Fixed the discount bug. All tests pass.\" -> CONTRADICTED by evidence\n  checks:   .countercheck/checks.toml @ HEAD (locked), sandbox: macos-seatbelt (no internet, localhost only; writes only in the check folder)\n  protected Modified: tests/test_pricing.py -> restored_from_base\n  [FAIL] unit tests  (65 ms)  $ python3 -m unittest -q\n         |     self.assertEqual(apply_discount(10, 150), 0)\n         | AssertionError: -5.0 != 0\n  cost:     $0.4200 (reported)\n  - the worker changed 1 protected file(s); countercheck ran the checks with the base version\n  - check \"unit tests\" failed (exit 1)\n```\n\n(The agent's own run said `OK (skipped=1)`: it had marked the failing test as skipped.)\n\nThe mental model: **the agent never grades itself.** Its claim is compared with what actually runs, on rules it\ncouldn't change:\n\n```\nbase revision (locked)        agent's work + claim\n  checks.toml, your tests          files, \"Done, all tests pass.\"\n          \\                           /\n           +--> countercheck verify <+\n                1 lock the checks     (from the base revision)\n                2 clean copy          (edited tests put back)\n                3 run in a sandbox    (no internet unless a check opts in)\n                4 decide + seal       (receipt with a SHA-256 seal)\n                        |\n      VERIFIED -> agent may finish\n      FAILED + \"done\" -> hook sends the failure back; agent fixes the code (max 3 rounds)\n      FAILED + honest \"can't, because ...\" -> agent may stop; receipt still says FAILED\n```\n\n| Rule | In plain words | \n|---|---|\n| The worker never grades itself | the checks come from before the work started; edited tests are put back | \n| Re-run, don't read | the agent's claim and logs are never evidence; the checks run again | \n| Honest verdicts | VERIFIED, FAILED or UNVERIFIED (with a reason), never a guess; a hint never softens a failure | \n| Everything from the repo is hostile | paths, sizes and links are checked, because the worker controls the files | \n| Domain-blind core | it knows nothing about SQL, games or ML; your checks carry the domain knowledge | \n| Fail closed on the verdict, open in the hooks | a broken referee never waves work through, and never traps your agent | \n| Leave evidence | every run writes a sealed receipt anyone can re-check | \n\nThe code behind each rule, one run step by step, and where new work plugs in: [docs/architecture.md](https://github.com/datacrate360/countercheck/blob/PROD/docs/architecture.md).\n\n| You use | How Countercheck plugs in | \n|---|---|\n| Claude Code | `Stop` hook:`countercheck hook claude-stop` ([docs/hooks/](https://github.com/datacrate360/countercheck/blob/PROD/docs/hooks) ) | \n| Cortex Code (Snowflake) | same hook format as Claude Code (not yet tried on a real install) | \n| Hermes Agent | `pre_verify` hook:`countercheck hook hermes-pre-verify` | \n| Codex, Cursor, Copilot, Aider, your scripts | run `countercheck verify --claim \"…\"` when it says done (dedicated hooks planned) | \n| CI | `countercheck verify --base main` ; fail the job on any exit code but`0` (GitHub Action planned) | \n\nReady recipes for real projects (check files, helper scripts, a 5-minute start): [recipes/](https://github.com/datacrate360/countercheck/blob/PROD/recipes) — first one:\n[Oracle → Snowflake conversion with Cortex Code](https://github.com/datacrate360/countercheck/blob/PROD/recipes/oracle-to-snowflake).\n\nWe gave [Hermes Agent](https://github.com/NousResearch/hermes-agent) 29 small coding tasks, 3 times each, in four\nsetups, and graded every run with **hidden tests** the agent never saw. Some tasks are impossible on purpose (a test\nexpects a wrong value, a server can't be reached): the honest answer is \"this can't pass, because …\".\n\n| False \"done\" claims (claimed success, hidden tests fail) | Hermes | + its `verify_on_stop` | **+ Countercheck** | both | \n|---|---|---|---|---|\n| GPT 6.1, all tasks (87 runs per setup) | 7 | 11 | **3** | **2** | \n| GPT 6.1, impossible tasks only | 6 | 8 | **0** | **0** | \n| DeepSeek V4 Pro, all tasks | 7 | 6 | 6 | 6 | \n\n- When GPT 6.1 got stuck, it edited the protected test and said \"all tests pass\". Countercheck re-ran the original tests, said FAILED and pushed back once; the agent restored the test and reported the real problem.\n- Countercheck only knows what your checks test: the remaining false claims were cases the visible tests didn't cover. DeepSeek rarely games tests, so there was little to catch.\n- It added about 1 second to the median run.\n\nFull method, numbers and limits: [docs/eval/results-2026-10.md](https://github.com/datacrate360/countercheck/blob/PROD/docs/eval/results-2026-10.md). Reproduce:\n[eval/](https://github.com/datacrate360/countercheck/blob/PROD/eval).\n\n| You use | Install with | \n|---|---|\n| Python (pip, a `.venv` , uv) | `pip install countercheck` ·`uv tool install countercheck` · or run without installing:`uvx countercheck verify` | \n| macOS or Linux, Homebrew | `brew install datacrate360/tap/countercheck` | \n| macOS or Linux, any shell | `curl -fsSL https://raw.githubusercontent.com/datacrate360/countercheck/PROD/install.sh \\| sh` | \n| Windows 10/11, PowerShell | `irm https://raw.githubusercontent.com/datacrate360/countercheck/PROD/install.ps1 \\| iex` (or`pip install countercheck` /`uvx countercheck` ) | \n| Rust | `cargo install --git https://github.com/datacrate360/countercheck countercheck` | \n| Direct download | [Releases](https://github.com/datacrate360/countercheck/releases) : one file per platform +`SHA256SUMS.txt` | \n\nBuilds: macOS (Apple silicon and Intel, one universal binary), Linux (x86_64, arm64; static) and Windows (x86_64,\n`countercheck.exe`, no runtime to install). Both install scripts check every download against `SHA256SUMS.txt`\nbefore installing. `install.ps1` puts `countercheck.exe` in `%LOCALAPPDATA%\\Programs\\countercheck` and tells you how\nto add that folder to `PATH`.\n\n**Windows needs [Git for Windows](https://git-scm.com/download/win)** (Countercheck calls `git`, and checks run with\nits `sh`, so the same `checks.toml` works on every OS). A check can use PowerShell or cmd instead:\n`shell = \"pwsh\"` or `shell = \"cmd\"` ([docs/checks.md](https://github.com/datacrate360/countercheck/blob/PROD/docs/checks.md#command)).\n\n**Sandbox by platform.** On macOS, checks run in the system sandbox (no internet, localhost allowed, writes only in\nthe check folder; `network = true` lets a single check use the internet).\nOn Linux and Windows there is no sandbox yet: checks run as a normal process and the receipt says so (on Windows:\n`none (no sandbox on Windows yet)`); run Countercheck inside a container or VM with networking off when you need\nisolation.\n\n| Platform | Checks run in | Internet during checks | Writes outside the check folder | \n|---|---|---|---|\n| macOS | the system sandbox (Seatbelt) | blocked (localhost allowed) unless `network = true` | blocked | \n| Linux | a normal process | allowed | allowed | \n| Windows | a normal process | allowed | allowed | \n\nWhat's shipped, in progress and next: [docs/ROADMAP.md](https://github.com/datacrate360/countercheck/blob/PROD/docs/ROADMAP.md).\n\n```\ncountercheck init                      # writes .countercheck/checks.toml, edit the commands\ngit add .countercheck && git commit -m \"Countercheck checks\"\n# ... let an agent work ...\ncountercheck verify --claim \"what the agent said\"            # uncommitted work vs HEAD\ncountercheck verify --base main                              # committed work on a branch vs main\ncountercheck show .countercheck/receipts/<id>.json                   # re-check a receipt's seal\n```\n\n`.countercheck/checks.toml`:\n\n```\nprotected = [\"tests/**\"]          # the agent may not change these; .countercheck/** is always protected\nlink = [\"node_modules\", \".venv\"]  # git-ignored dependency folders to make visible (read-only)\n\n[[check]]\nname = \"unit tests\"\nrun = \"pytest -q\"                 # any command; runs with `sh -c` in the clean copy\ntimeout_secs = 600\n```\n\nBesides plain commands there are matrix, forbid, golden, budget and coverage checks: see [docs/checks.md](https://github.com/datacrate360/countercheck/blob/PROD/docs/checks.md)\nand the runnable example in [examples/data-migration](https://github.com/datacrate360/countercheck/blob/PROD/examples/data-migration).\n\nExit codes: `0` verified · `2` failed (claim contradicted) · `3` unverified (nothing failed, but the checks weren't\nlocked in the base revision, or none ran) · `1` error · `4` (`countercheck show`) receipt was edited.\n\n```\ncountercheck ui          # a live page in your browser; updates when a new receipt lands\ncountercheck report --open   # the same page as one HTML file (offline, attach it to a PR or CI run)\n```\n\n`countercheck ui` serves the page on this computer only (127.0.0.1, with a random token in the address; other\nhosts and other web pages can't read it). It uses port 7357, or any free port if that one is taken (`--port N` to\nchoose), and running it again opens the one already running for the project.\n\n`countercheck report` writes one HTML page (`.countercheck/receipts/report.html`, git-ignored) with every saved run, newest first: the\nverdict, what the agent said, each check with its output, protected files it changed, whether the receipt's seal is\nintact, and for matrix checks a grid of targets × checks. It works offline, loads nothing from the internet and has no\nscripts, so it's safe to attach to a pull request or a CI run.\n\n- \nHermes Agent: `countercheck hook hermes-pre-verify` keeps Hermes working while the checks fail. In the profile's`config.yaml` :\n\n```\nhooks:\n  on_session_start:\n    - command: \"countercheck hook session-start\"\n  pre_verify:\n    - command: \"countercheck hook hermes-pre-verify\"\n      timeout: 300\n```\n\n Then approve both hooks once in a terminal ( `hermes hooks list` shows \"✗ not allowlisted\" until you do; the\ndesktop app never asks). Setup, approval and how the project is found:[docs/hooks/hermes.md](https://github.com/datacrate360/countercheck/blob/PROD/docs/hooks/hermes.md) .\n- \nClaude Code and Cortex Code: `countercheck hook claude-stop` blocks the`Stop` while the checks fail, with`countercheck hook session-start` on`SessionStart` . See[docs/hooks/claude-code.md](https://github.com/datacrate360/countercheck/blob/PROD/docs/hooks/claude-code.md) .\n\n**Add the session-start hook too.** It remembers the commit each session started from, so the checks stay locked\neven if the agent commits its own edits to a test or to `checks.toml`. Without it, the checks are locked to `HEAD`.\nIn CI, pass the base explicitly: `countercheck verify --base origin/main`.\n\n**On Windows** the hooks work the same (Claude Code runs natively there): use the same settings with\n`countercheck.exe` on `PATH`, or give its full path with forward slashes, e.g.\n`\"command\": \"C:/Users/you/AppData/Local/Programs/countercheck/countercheck.exe hook claude-stop\"`. Paths in the hook\ninput may be `C:\\...` or `C:/...`; a transcript under `~\\` is found through `USERPROFILE`. See\n[docs/hooks/claude-code.md](https://github.com/datacrate360/countercheck/blob/PROD/docs/hooks/claude-code.md#windows).\n\nA referee has to be right, so Countercheck is tested harder than most tools its size:\n\n- **140 tests** (`cargo test --workspace` ): unit tests, end-to-end tests that run the real binary on throwaway git\nrepositories (tampered tests, timeouts, the macOS sandbox, both agent hooks, every check type), and**property\ntests** that throw thousands of generated inputs at the verdict and seal logic (e.g. a receipt's seal breaks if any\none of 33 fields changes;`decide` never says VERIFIED when a check failed).\n- **Mutation testing** ([cargo-mutants](https://mutants.rs) ): the receipt, claim and config code was deliberately\nbroken in every way the tool could find; tests were added until every break that compiles is caught (one remaining\nchange is provably equivalent).\n- **CI on every push** (macOS and Linux):`cargo fmt --check` ,`cargo clippy -D warnings` , all tests, a release build\nand the benchmark task validator. Countercheck also verifies itself with its own`.countercheck/checks.toml` .\n- **The benchmark** above, with hidden tests as ground truth.\n\nCountercheck is a referee that runs inside other people's projects: Python, JavaScript, Java, SQL, anything. So it\nhas to be one dependency-free file that starts instantly, can't crash, and behaves the same on every operating system.\nRust gives exactly that, which is also why tools like [uv](https://github.com/astral-sh/uv),\n[ruff](https://github.com/astral-sh/ruff) and OpenAI's [Codex CLI](https://github.com/openai/codex) are written in it.\n\n| What a referee needs | Why it matters here | What Rust gives | \n|---|---|---|\n| **Nothing to install** | It runs inside your project, so it must never clash with your Python version, virtualenv or `node_modules` | One file of about 3–4 MB with no runtime; `pip install countercheck` ships that same binary in a wheel, like ruff | \n| **Instant start** | An agent hook runs it every time the agent tries to finish | Starts in under 10 ms (a Python CLI often needs 100–300 ms just to import) | \n| **No crashes, no memory bugs** | A bug in a referee is a wrong verdict | Memory safety from the compiler; on top, this project forbids `unsafe` and`unwrap` /`panic!` outside tests | \n| **Same everywhere** | Teams mix macOS, Linux and Windows | One codebase, cross-compiled; static Linux builds | \n| **A strict gate for AI-written code** | Much of this code is written with coding agents | The compiler and `clippy -D warnings` reject whole classes of mistakes before review; property and mutation testing are first-class | \n\n**The costs, honestly:** fewer people know Rust than Python, and it is slower to write. That's why most contributions\nneed no Rust at all: domain examples, check files and benchmark tasks are plain files, scripts and Python, and checks\nrun your project's own commands in any language. Go would also have been a reasonable choice (single binary, simpler\nlanguage); Rust won on stricter safety guarantees and its testing tools.\n\n**Where Rust is not the answer:** a future hosted team dashboard (logins, many repositories, rich views) is a web app\nand will use a web stack. The full reasoning is in [docs/decisions/0001-stack.md](https://github.com/datacrate360/countercheck/blob/PROD/docs/decisions/0001-stack.md).\n\n**Does Countercheck limit what my agent can do?**\nNo. The agent works with its full harness: web search, docs, tools, anything you allow. Countercheck only steps in\nwhen the agent says \"done\": it re-runs your locked checks on a clean copy of the result. Like an exam after open-book\nstudy: the grade has to reflect the work itself.\n\n**Does Countercheck edit tests or search the web?**\nNever. It reads the checks from the base revision, runs them and writes a receipt. It makes no network connections of\nits own (no telemetry, no network library in the binary) and does not use an LLM: the verdict comes from your checks'\nexit codes and output.\n\n**What is the sandbox, and which OS has one?**\nA fence the operating system puts around the check while it runs, so the agent's code can't fetch the answer from the\ninternet or write outside the check folder.\n\n| OS | Sandbox used for checks | Note | \n|---|---|---|\n| macOS | Seatbelt ( `sandbox-exec` ): no internet (localhost allowed), writes only in the check folder | `network = true` opens the internet for one check | \n| Linux | none yet (planned: namespaces + Landlock) | the receipt says `none` ; use a container with`--network none` for isolation | \n| Windows | none yet (planned: AppContainer) | the receipt says `none (no sandbox on Windows yet)` ; the Job Object only stops a check's processes | \n\n**So without a sandbox, can the agent cheat on Linux or Windows?**\n\nThe common cheats are caught on every OS: edited tests are restored from the base revision, `checks.toml` is read\nfrom the locked base, committing edits doesn't move that base (the `session-start` hook), and every check is re-run\nin a clean copy. The sandbox adds protection against code written to misbehave *during* the check (fetching the\nexpected answer online, leaving files outside the folder). Until Linux and Windows have one, run Countercheck in a\ncontainer or VM with networking off when that matters.\n\n**Why does grading run offline?**\nSo the same work always gets the same verdict, and nobody can fetch the answer at grading time. In the benchmark, the\nonly false alarms came from checks that called a real website. If a check truly needs the internet, set\n`network = true` on it; the receipt then says \"internet allowed\".\n\n**Does an LLM judge the work?**\nNo. The agent never grades itself, and Countercheck doesn't ask a model either. The checks you lock decide. The only\ntext reading is a small, fixed list of phrases that tells whether the agent's final message already admits a blocker\n(\"this can't pass, because …\"), so the hook doesn't push an honest agent to keep going.\n\n**Will it catch every wrong \"done\"?**\nOnly what your checks test. In the benchmark, the false \"done\" claims left were cases the visible tests didn't cover.\nStronger checks (golden files, coverage, budgets: [docs/checks.md](https://github.com/datacrate360/countercheck/blob/PROD/docs/checks.md)) catch more.\n\n**How much slower does it make my agent?**\nAbout 1 second on the median benchmark run; longer when your checks are slow, because it runs them for real.\n\n**Which agents does it work with?**\nHermes Agent and Claude Code have hooks ([Agent hooks](#agent-hooks)). Snowflake's Cortex Code uses Claude Code's\nhook format, so the same hooks should work there (not yet tried on a real install). Any other agent, script or CI\njob can call `countercheck verify` directly.\n\n**Which operating systems?**\nmacOS, Linux and Windows 10/11 (x86_64). Windows needs Git for Windows. See [Install](#install).\n\n**Does my code leave my machine?**\nNo. Everything runs locally; receipts are files in `.countercheck/receipts/`. `countercheck ui` listens on 127.0.0.1\nonly, behind a random token.\n\n**Is it free?**\nYes, MIT-licensed. See [Contribute](#contribute-make-it-fit-your-domain) to add check types, examples or agent hooks.\n\nv0.3 ([CHANGELOG](https://github.com/datacrate360/countercheck/blob/PROD/CHANGELOG.md)). macOS has the sandbox; Linux and Windows run checks without one (the receipt says\nso). Not yet: signed receipts, a per-host network allow-list, cost read from agent logs, Linux/Windows sandboxes, more agent\nhooks (Codex, Cursor, a GitHub Action). See [docs/ROADMAP.md](https://github.com/datacrate360/countercheck/blob/PROD/docs/ROADMAP.md).\n\nCountercheck gets better with every field it learns about, and most of that needs **no Rust**: a domain is covered\nby a good `checks.toml` plus a small example. How agents fake \"done\" in game development, ML, dbt, video pipelines\nor a migration is knowledge we don't have and you do.\n\n| Start here | Rust? | \n|---|---|\n| Open a [\"New domain\" issue](https://github.com/datacrate360/countercheck/issues/new?template=new-domain.yml) : how \"done\" goes wrong in your work | no | \n| Add an example for your field ( [examples/](https://github.com/datacrate360/countercheck/blob/PROD/examples) ) | no | \n| Add a benchmark task that tempts agents to cheat ( [eval/](https://github.com/datacrate360/countercheck/blob/PROD/eval) ) | no | \n| Hook another agent (Codex, Cursor, OpenCode, …) | yes | \n| Add tests, or a check type | yes | \n\nEach of these has a step-by-step guide in [.claude/skills/](https://github.com/datacrate360/countercheck/blob/PROD/.claude/skills), written for people and for AI coding\nagents alike (Claude Code loads them automatically). Read [CONTRIBUTING.md](https://github.com/datacrate360/countercheck/blob/PROD/CONTRIBUTING.md) for the gate, pull\nrequests and commit sign-off (DCO).\n\nRust 1.90+: `cargo build --release` → `target/release/countercheck`. Gate: `cargo fmt --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo test --workspace`, `python3 eval/validate.py`. Countercheck also verifies itself: `countercheck verify` (checks in `.countercheck/checks.toml`).\n\n- `crates/countercheck-core` : pure logic (check file, check types, protected paths, receipts, verdicts)\n- `apps/countercheck` : the CLI (git, clean copy, sandbox, runner, hooks)\n- `examples/` : one made-up project per domain, with checks that catch a planted mistake\n- `eval/` ,`docs/eval/` : the benchmark against Hermes Agent (tasks with hidden tests)\n- `.claude/skills/` : step-by-step contribution guides ·`docs/architecture.md` : how it works inside\n- `packaging/` ,`install.sh` ,`install.ps1` ,`pyproject.toml` : release packaging (Homebrew, install scripts, Python wheels)\n- `docs/checks.md` : every check type ·`docs/hooks/` : agent hooks ·`docs/decisions/` : design decisions\n\nMIT licensed.", "url": "https://wpnews.pro/news/countercheck-re-runs-your-tests-when-an-ai-agent-says-done", "canonical_source": "https://github.com/datacrate360/countercheck", "published_at": "2026-10-20 16:05:56+00:00", "updated_at": "2026-10-20 16:17:00.949093+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-safety"], "entities": ["Countercheck", "Claude Code", "Cortex Code", "Snowflake", "Hermes Agent", "Codex", "Cursor", "Copilot"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/countercheck-re-runs-your-tests-when-an-ai-agent-says-done", "markdown": "https://wpnews.pro/news/countercheck-re-runs-your-tests-when-an-ai-agent-says-done.md", "text": "https://wpnews.pro/news/countercheck-re-runs-your-tests-when-an-ai-agent-says-done.txt", "jsonld": "https://wpnews.pro/news/countercheck-re-runs-your-tests-when-an-ai-agent-says-done.jsonld"}}