# Trustbut checks if an AI agent's new tests fail without its change

> Source: <https://github.com/isAlmogK/trustbut>
> Published: 2026-10-18 19:07:31+00:00

**Your agent says it's done and the tests pass. Does anything actually prove the change?**

🧾 *Verified by trustbut: pull requests to this repo carry its own trustbut receipt.*

[Languages](#supported-languages) · [Why](#why-im-building-this) · [What we've measured](#what-weve-measured) · [Install](#install) · [Usage](#usage) · [Use it with your agent](#use-it-with-your-agent) · [Config](#configuration-trustbutjson) · [What it checks](#what-it-checks) · [Receipt format](#the-agent-receipts-format) · [CI](#use-it-in-ci) · [How it works](#how-it-works) · [FAQ](#faq) · [Roadmap](https://github.com/isAlmogK/trustbut/blob/main/docs/ROADMAP.md) · [Sponsor](#sponsor) · [About me](#about-me)

trustbut is a local command-line tool that produces a **receipt** for a code change, whether an AI agent or a person wrote it. It reruns your tests in a clean checkout, runs every new or changed test against the code from before the change to see whether it fails there (if it passes either way, it doesn't test the change), and flags skipped tests, removed assertions and silenced type checks. Then it gives you a **PASS / WARN / FAIL** verdict. Wired into your agent, it can also stop the agent from finishing while the receipt is FAIL, so it keeps fixing until the evidence holds up ([details](#use-it-with-your-agent)).

It works with **any agent** (Claude Code, Codex, Cursor, Gemini CLI, OpenCode, Copilot) and **11 languages** (JavaScript/TypeScript, Python, Go, Rust, Java, Kotlin, C#, Ruby, PHP, Dart/Flutter and Swift) because it only needs git and your test runner. No LLM, no server, no account. Your code never leaves your machine.

``` bash
$ npx trustbut verify

## 🧾 Receipt: WARN

`3f2a91c` → `8be04d1` · runner: vitest

**Tests (clean checkout):** 41 passed · 0 failed · 1 skipped
**Proof:** 1 new/changed test proves the change ✅
**New/changed tests:** 1 prove the change · 1 pass without it

### Findings
- ⚠ **Test skipped** · `test/auth.test.ts:88`: it.skip('handles clock skew', () => {
- ⚠ **cast to any** · `src/auth/session.ts:41`: const claims = decoded as any;

### Review these first
1. `src/auth/session.ts`: auth/permissions, has findings
2. `package.json`: dependencies
```

When the suite fails at head, the receipt also lists the failing tests. If `package.json` declares dependencies that aren't installed in your `node_modules` (a stale install), a `deps-not-installed` warning names them, because both runs link and use your installed dependencies. See the finding ids in the spec's [finding catalog](https://github.com/isAlmogK/trustbut/blob/main/spec/agent-receipt-v1.md#finding-catalog).

The output is Markdown, so the same receipt reads cleanly in your terminal and renders properly in a pull request or a CI job summary.

| Language | Test frameworks | Tamper and escape checks | Verified in CI against the real toolchain | 
|---|---|---|---|
| JavaScript / TypeScript | Vitest, Jest (incl. Expo / React Native), Mocha, `node:test` | ✅ | ✅ | 
| Python | pytest | ✅ | ✅ | 
| Go | `go test` | ✅ | ✅ | 
| Rust | cargo-nextest, `cargo test` | ✅ | ✅ | 
| Java | Maven, Gradle | ✅ | ✅ | 
| Kotlin | Maven, Gradle | ✅ | ◐ same runners as Java; Kotlin end-to-end test coming | 
| C# / .NET | `dotnet test` (xUnit verified; NUnit and MSTest via the same TRX results) | ✅ | ✅ | 
| Ruby | RSpec | ✅ | ✅ | 
| PHP | PHPUnit | ✅ | ✅ | 
| Dart / Flutter | `dart test` ,`flutter test` | ✅ | ✅ | 
| Swift | Swift Package Manager (XCTest, Swift Testing) | ✅ | ✅ | 
| Anything else | any command that writes JUnit XML | — | ✅ | 

trustbut detects the right runner automatically, including in monorepo subfolders. Setup steps (such as starting a database emulator), extra arguments and environment variables go in a [`.trustbut.json`](#configuration-trustbutjson). Detection rules and the exact commands are in [Supported languages and test runners](#supported-languages-and-test-runners).

AI agents now write a large share of the code I ship. Writing it is no longer the slow part. **Checking it is.**

When an agent says "done, all tests pass", that's the agent reporting on its own work. Reading the diff doesn't show what actually ran, or whether the new tests would fail if the change were wrong. I wanted a check from outside the agent that answers those two questions with evidence, not with another model's opinion.

The data says I'm not alone:

- In the [2026 Stack Overflow survey](https://survey.stackoverflow.co/2026/ai) ,**48%** of developers trust AI output when they can easily validate it, but only**16%** trust it for most tasks. The gap between those numbers is the problem trustbut works on.
- In 2025, **66%** of developers named "almost right, but not quite" as their top frustration with AI tools ([InfoWorld on the 2025 survey](https://infoworld.com/article/4031673/ai-use-among-software-developers-grows-but-trust-remains-an-issue-stack-overflow-survey.html) ).
- Armin Ronacher put it well in [The Final Bottleneck](https://lucumr.pocoo.org/2026/2/13/the-final-bottleneck/) : generating code is now cheaper than reviewing it, and the human stays accountable.

The labs have also reported their models gaming tests in agentic coding:

- Anthropic's [Claude 3.7 Sonnet system card](https://www.anthropic.com/claude-3-7-sonnet-system-card) reports the model special-casing tests, and sometimes modifying them, to make them pass in agentic coding environments like Claude Code. The[Claude 4 system card](https://www.anthropic.com/claude-4-system-card) reports this much reduced, but still present.
- OpenAI's [research on monitoring reasoning models](https://arxiv.org/abs/2503.11926) caught a model exiting early or skipping unit tests to look green.
- [METR](https://metr.org/blog/2025-06-05-recent-reward-hacking/) found frontier models patching the grading and scoring code of their evaluations, and[ImpossibleBench](https://arxiv.org/abs/2510.20270) measures how often models exploit tests on tasks that can't be solved honestly.

That's why trustbut also flags skipped tests, removed assertions and silenced checks. But in my own measurement of 201 real pull requests, merged agent work rarely showed these behaviours (see [What we've measured](#what-weve-measured)). The question that holds up is the plainer one: does anything prove this change?

AI code reviewers read the diff and comment, but they don't rerun anything. A vendor checking its own agent is a weak audit, and most teams use more than one agent. So trustbut is **independent, deterministic and local**: it doesn't ask an AI whether the code looks right. It runs things and reports what it found.

I'm measuring whether trustbut is useful, in the open, in [trustbut-index](https://github.com/isAlmogK/trustbut-index). It runs trustbut on real merged pull requests from established open-source projects: PRs written by agents (Copilot, Codex, Claude Code, Cursor, Devin, Jules) and, as a control, PRs written by people in the same projects. These are pilot results, not the final study.

**trustbut 0.7.0 on 201 pull requests:**

|  | Agent PRs | Human PRs | 
|---|---|---|
| trustbut could run the tests | 53 of 103 | 56 of 98 | 
| A test that passed before the change fails after it¹ | 0 of 53 | 0 of 56 | 
| New or changed tests that fail without the change, so they test it | 275 of 283 (97%) | 294 of 333 (88%) | 
| PRs with any finding | 12 of 53 | 15 of 56 | 
| PRs where nothing proves the change ( `change-not-proven` ) | 6 of 53 | 7 of 56 | 

¹ 18 of the 109 runs had a failing test with no result from before the change, so this count is a lower bound.

What this says so far:

- **Agents didn't break suites or fake their tests in these PRs.** Their new tests were at least as likely as people's to test the change. Merged PRs have already been through CI and review, though, so this says little about what an agent does before it says "done". That's where the[agent hook](#use-it-with-your-agent) works, and it isn't measured yet.
- **Most findings are true but don't need action.** In the 0.6.0 pilot, 71 findings were labeled: 5 were worth a reviewer acting on (7%), 57 were true but harmless, and 9 were wrong. All 5 useful ones said that nothing proved the change. 0.7.0 fixed the causes of the 9 wrong findings and made that one its headline,`change-not-proven` . The labels are drafts by Claude that no person has confirmed yet, and the labeler could see the code, so they weren't fully blind.
- **trustbut ran on about half of the PRs.** Of the 92 it couldn't run, 78 failed to install the project's dependencies or toolchain in the study's runner, 13 had no supported test runner, and 1 result is missing.

**Not measured yet:** whether `change-not-proven` findings are worth acting on, and whether the agent hook leads to better agent work.

On my own repos ([DogBase](https://www.dogbase.co)), receipts found a fix commit with no test, 7 new tests that passed with or without their change, two new `eslint-disable` lines, and a dependency missing from the local install, which became the `deps-not-installed` warning.

Have a run of your own? [Open an issue](https://github.com/isAlmogK/trustbut/issues) with what the receipt found, or missed.

**Requirements:** Node.js 22 or newer, git, and a repository with tests in one of the [supported languages and test runners](#supported-languages-and-test-runners), or any runner that can write JUnit XML. The toolchain of your project (Go, Cargo, Maven, .NET, Composer, …) must be installed, as it is when you run the tests yourself.

**Run without installing (recommended):**

```
npx trustbut verify
```

**Install globally:**

```
npm i -g trustbut
trustbut verify
```

**From source:**

```
git clone https://github.com/isAlmogK/trustbut.git
cd trustbut
npm ci
npm link
```

To have your agent verify its own work after every turn, see [Use it with your agent](#use-it-with-your-agent).

Run it in your repository after an agent finishes:

```
trustbut verify            # the committed work on your branch
trustbut verify --working  # uncommitted work too (tracked edits plus untracked files)
```

By default trustbut compares your branch with the point where it split off `main` (or `master`), the same way a GitHub pull request shows its diff.

```
# Compare against a specific branch or commit
trustbut verify --base develop
trustbut verify --base HEAD~3

# Verify uncommitted work (the agent hasn't committed yet)
trustbut verify --working

# Faster: rerun only the new and changed test files at head
trustbut verify --quick

# The tests live in a subfolder, or the config lives outside the repo
trustbut verify --root services/api
trustbut verify --config ~/configs/myapp.trustbut.json

# Gate a script or CI job on warnings as well as failures
trustbut verify --strict

# Machine-readable output (an agent receipt v1)
trustbut verify --json > receipt.json
```

| Option | Description | Default | 
|---|---|---|
| `-b, --base <ref>` | Base branch or commit, compared from its merge-base with head | `origin/main` ,`main` ,`origin/master` or`master` | 
| `--head <ref>` | The commit to verify (not with `--working` ) | `HEAD` | 
| `--working` | Verify uncommitted work: tracked edits plus untracked, non-ignored files, snapshotted without touching your index | off | 
| `--runner <name>` | Force a test runner, any name from the [table below](#supported-languages-and-test-runners) | detected automatically | 
| `--root <dir>` | Package folder to run the tests in, relative to the repo root (overrides `root` in the config) | `.` | 
| `--config <path>` | Use this config file instead of `.trustbut.json` at head; it may live outside the repo | `.trustbut.json` | 
| `--quick` | At head, rerun only the new and changed test files | off (full suite) | 
| `--strict` | Exit with `1` on WARN as well as FAIL | off | 
| `--json` | Print the receipt as JSON: an [agent receipt v1](https://github.com/isAlmogK/trustbut/blob/main/spec/agent-receipt-v1.md) | off | 
| `--receipt-json <path>` | Also write the JSON receipt to this file (relative to the current directory; parent folders are created) | off | 
| `--output-tail` | JSON receipts: add the end of each failed run's output (last 50 lines, at most 4,000 characters) as `outputTail` under`x-trustbut` . Tests can print secrets | off | 
| `--receipt-md <path>` | Also write the Markdown receipt to this file | off | 

| Verdict | Meaning | Exit code | 
|---|---|---|
| ✅ **PASS** | Tests are green from a clean checkout, the new tests prove the change, and nothing suspicious was found | `0` | 
| **WARN** | Something deserves a human look: a change no test proves, a skipped test, a silenced check | `0` (`1` with`--strict` ) | 
| ✖️ **FAIL** | Tests fail from a clean checkout, a changed test fails, or a test was focused ( `.only` ) | `1` | 
| — | Usage or runtime error, e.g. an unknown ref or option, or a `--receipt-json` /`--receipt-md` file that couldn't be written | `2` | 

Tip

By default trustbut only looks at **committed** changes. If you have uncommitted work, the receipt says so. Commit first, or pass `--working` to include it.

trustbut can run itself at the end of every agent turn, so a FAIL receipt never reaches you as "done".

Note

The commands below assume `trustbut` is installed globally (`npm i -g trustbut`).

**What happens**

1. When the agent finishes a turn, trustbut quickly verifies its work: your uncommitted changes plus the commits on your branch. By default it reruns only the new and changed test files (the whole suite when none changed).
2. **FAIL:** the agent gets the receipt (failing tests with their messages, focused tests, and so on) and keeps fixing. This repeats up to`hook.maxRetries` times in a row (default 3), then the agent is allowed to stop and you see the FAIL.
3. **WARN or PASS:** you get a one-line note and the agent finishes.
4. trustbut never blocks on its own errors: a timeout, a crash or a missing tool lets the agent finish with a "couldn't verify" note.
5. A tree already verified isn't rerun. A FAIL is sent back once per tree: if the agent stops again without changing anything, or you start a new turn with nothing changed, it is allowed to finish and you see the FAIL with the receipt path.

After each completed check, the full Markdown receipt is at `.git/trustbut/last-receipt.md` (per worktree; inside `.git`, so it is never committed).

Things to know:

- The hooks run on macOS and Linux (POSIX) for now; Windows isn't supported yet.
- Use either the Claude Code plugin or `trustbut init claude` , not both: with both, every turn is checked twice.
- The hook runs your tests outside the agent's sandbox; see [Is it safe to run?](#faq) .

Install the plugin from this repository's marketplace:

```
/plugin marketplace add isAlmogK/trustbut
/plugin install trustbut@trustbut
```

Or write the hook into your project without the plugin: `trustbut init claude` (use one or the other, not both). The plugin uses the `trustbut` on your PATH when it is recent enough, and otherwise runs `npx trustbut@<version>`. Both run the hook in exec form (` args`), which needs a recent Claude Code (2.1 or later).

When trustbut sends a FAIL back, Claude Code labels it "Stop hook error occurred". This is expected: it is how Claude Code shows a blocking Stop hook, and the receipt is still delivered to the agent.

```
trustbut init codex
```

Codex asks you to approve new hooks: open `/hooks` and approve **trustbut verify** (needs Codex 0.129 or later). Project hooks load only in trusted projects.

```
trustbut init gemini
```

Project hooks run only in trusted folders, so trust the folder in Gemini CLI if the hook doesn't fire. Needs Gemini CLI 0.35 or later.

```
trustbut init opencode
```

This writes a plugin for the OpenCode V1 plugin API (`opencode-ai` 1.x; V2 support is a follow-up). OpenCode can't block "done", so on a FAIL the receipt is sent back to the agent as a follow-up prompt instead.

`trustbut init <claude|codex|gemini|opencode>` copies a small runner script next to the agent's config and, for Claude Code, Codex and Gemini CLI, merges one hook entry into it, leaving your other hooks alone. For OpenCode it writes a plugin file instead. It is safe to rerun: it updates its own entry in place.

| Option | What it does | 
|---|---|
| `--dry-run` | Print what would change and write nothing | 
| `--user` | Set up your user-level config ( `~/.claude` ,`~/.codex` ,`~/.gemini` ,`~/.config/opencode` ) instead of the project | 

Project files are written at the git root, and agents read project settings from the directory they were started in, so start the agent from the repo root. A settings file that contains comments is refused (exit `2`) with the snippet to add by hand.

Set `TRUSTBUT_SKIP=1` in the environment to skip the check for a session (for example `TRUSTBUT_SKIP=1 claude`). To change how it behaves, add a `hook` object to [`.trustbut.json`](#configuration-trustbutjson). It is read as committed at `HEAD`, so an uncommitted edit by the agent can't switch it off.

```
{ "hook": { "mode": "full", "maxRetries": 2 } }
```

| Key | Type | Default | Description | 
|---|---|---|---|
| `enabled` | boolean | `true` | `false` lets every stop through without verifying | 
| `mode` | `"quick"` or`"full"` | `"quick"` | `quick` reruns only the new and changed test files at head;`full` reruns the whole suite | 
| `maxRetries` | whole number, 0 to 100 | `3` | How many times in a row a FAIL is sent back to the agent. `0` reports a FAIL without ever blocking | 
| `timeoutSec` | number above 0 | `540` | Time budget for one check, kept under the agents' 600 s hook timeout. When it runs out the agent is allowed to finish | 
| `base` | ref name | origin's default branch, then `origin/main` ,`main` ,`origin/master` ,`master` , then the branch's upstream, else`HEAD` | What to compare against | 

The hook protocols, exit codes and environment variables are in the [spec](https://github.com/isAlmogK/trustbut/blob/main/docs/SPEC.md#7-agent-hooks-trustbut-hook-agent).

trustbut picks the runner by looking at the package folder (the repo root, or `root` / `--root`) as committed at head (with `--working`, your working tree). If several match, the most specific one wins; force one with `--runner <name>` or `"runner"` in the config.

| Runner | Language | Detected by | Runs | Notes | 
|---|---|---|---|---|
| `vitest` | JS / TS | `vitest` dependency, or`vitest.config.*` /`vitest.workspace.*` | `vitest run --reporter=junit` | Wins over Jest when both are present | 
| `jest` | JS / TS | `jest` dependency or config,`jest.config.*` , or a`jest-expo` /`react-native` preset | `jest --ci --json --runTestsByPath <files>` | Works for Expo and React Native apps | 
| `mocha` | JS / TS | `mocha` dependency or`.mocharc.*` | `mocha --reporter json` |  | 
| `node-test` | JS / TS | `scripts.test` in`package.json` uses`node --test` | `node --test` | Uses the built-in `node:test` runner | 
| `pytest` | Python | `pytest.ini` ,`conftest.py` ,`tox.ini` , or`[tool.pytest` in`pyproject.toml` | `python -m pytest -q --junitxml` | Uses the checkout's `.venv` /`venv` if present | 
| `go` | Go | `go.mod` | `go test -json -count=1 ./...` | Add `"args": ["-p", "1"]` for packages that share a database or emulator | 
| `cargo` | Rust | `Cargo.toml` | `cargo nextest run` if installed, else`cargo test --no-fail-fast` | Also checks `#[cfg(test)]` modules in source files. Proofs come from integration tests under`tests/` : inline`#[cfg(test)]` tests run in the suite but give no fail-before / pass-after proof, since they live in the source file being changed | 
| `maven` | Java / Kotlin | `pom.xml` (the`mvnw` wrapper is preferred) | `mvn -B test` (Surefire reports) |  | 
| `gradle` | Java / Kotlin | `build.gradle(.kts)` or`settings.gradle(.kts)` (the`gradlew` wrapper is preferred) | `gradle test --continue` |  | 
| `dotnet` | C# | a `.sln` ,`.slnx` ,`.csproj` or`.fsproj` in the root | `dotnet test` (TRX results) | With several projects it prefers the solution file | 
| `rspec` | Ruby | `.rspec` , or`spec/` plus`rspec` in the`Gemfile` | `bundle exec rspec --format json` | Plain `rspec` when there is no Gemfile | 
| `phpunit` | PHP | `phpunit.xml(.dist)` , or`phpunit/phpunit` in`composer.json` | `vendor/bin/phpunit --log-junit` | Runs `composer install` in the temporary checkout, since`vendor/` isn't tracked | 
| `dart` | Dart | `pubspec.yaml` with a`test` dev dependency | `dart test --reporter json` | Runs `dart pub get` first when`.dart_tool` is missing | 
| `flutter` | Dart / Flutter | `pubspec.yaml` with the Flutter SDK | `flutter test --machine` |  | 
| `swift` | Swift | `Package.swift` | `swift test --parallel --verbose --xunit-output` | Swift Package Manager projects | 
| `command` | anything | never detected; chosen with `command` in the config | your command | See below | 

Each runner's own report (JUnit XML, JSON events or TRX, depending on the tool) is turned into the same per-test results, so each language gets the same fail-before / pass-after proof.

If your tests run through something not listed, point trustbut at a command that writes **JUnit XML**:

```
{
  "command": "./run-tests.sh {files}",
  "junit": "test/reports/junit.xml"
}
```

`{files}` is replaced with the quoted changed test files (and with nothing for a full run). Without `junit`, trustbut reads the command's stdout if it is JUnit XML. Setting `command` selects the `command` runner automatically.

Everything is optional; most projects need no config at all. Put `.trustbut.json` in the repo root. trustbut reads it **as committed at head** (so the version under review is the one used), then falls back to the working-tree file. `--config <file>` uses a different file instead, which can live outside the repo. Unknown keys and wrong types are errors (exit code `2`) that list the allowed keys; a `$schema` key is ignored.

| Key | Type | Default | Description | 
|---|---|---|---|
| `root` | string | `"."` | Package folder relative to the repo root. Runners are detected and run there, and only changed test files under it are rerun (changed tests elsewhere are listed as not run). Must stay inside the repo and exist at head (otherwise exit code `2` ).`--root` overrides it. | 
| `runner` | string | detected | Force a runner from the table above. `--runner` overrides it. | 
| `command` | string | none | The `command` runner: shell command to run.`{files}` becomes the quoted test files (nothing for a full run). | 
| `junit` | string | none | The `command` runner: JUnit XML file the command writes, relative to`root` . | 
| `setup` | string | none | Shell command started in the background before **each** test run (for example an emulator or database), and always killed afterwards. If it exits non-zero before the tests start, the run stops with exit code`2` . | 
| `setupReadyPattern` | regex string | none | Tests start as soon as `setup` prints a line matching this (stdout or stderr). | 
| `setupWaitSec` | number, 0 to 2,000,000 | `5` | How long to wait for `setup` ; with`setupReadyPattern` it is the maximum wait. | 
| `env` | object of strings | none | Extra environment variables for every test run and for `setup` . | 
| `args` | string[] | none | Extra arguments added to the runner's command: before the test files (or packages, selectors) for most runners; for Maven, Gradle and .NET at the end; for `swift` after the`--filter` options; for the`command` runner appended (shell-quoted) at the very end of the command. | 
| `timeoutSec` | number, up to 2,000,000 | `900` | Time limit for each test run; the receipt says when a run timed out. | 

**1. A Go backend with a Firebase emulator, in a subfolder**

```
{
  "root": "backend",
  "args": ["-p", "1"],
  "setup": "firebase emulators:start --only firestore,auth",
  "setupReadyPattern": "All emulators ready",
  "setupWaitSec": 90,
  "env": { "FIRESTORE_EMULATOR_HOST": "127.0.0.1:8080" }
}
```

`-p 1` runs packages one at a time so they don't fight over the emulator. The emulator is started before the head run and again before the base run, and always shut down.

**2. An Expo / React Native app with Jest**

```
{
  "runner": "jest",
  "args": ["--silent"]
}
```

Usually you don't need any config: `jest-expo` and `react-native` presets are detected. Dependencies are linked from your checkout, so there's no reinstall.

**3. Vitest in a `functions/` subfolder**

```
{ "root": "functions" }
```

Or skip the file and run `trustbut verify --root functions`.

**4. Any other tool, through the `command` runner**

```
{
  "command": "./scripts/run-tests.sh {files}",
  "junit": "build/test-results/junit.xml",
  "timeoutSec": 1800
}
```

Warning

`command` and `setup` are shell commands that run as part of verification, from the config at head, just like a changed `test` script in `package.json` would. Only verify code you'd be willing to run.

Your test suite runs in a fresh, temporary `git worktree` at head, not in your working directory and not by trusting the agent's own run. A red suite is a **FAIL**.

This is the core idea. Every new or changed test also runs against the **base** commit:

| Before the change | After the change | Result | 
|---|---|---|
| fails | passes | ✅ **Proves the change** | 
| passes | passes | **Passes without the change** : the test doesn't check the new behaviour. Listed in the receipt; the receipt warns only when no new or changed test proves the change | 
| — | fails | ✖️ **Fails at head** | 
| — | skipped | **Skipped** | 
| couldn't run (e.g. imports code that didn't exist yet) | passes | ✅ Counted, but marked as *couldn't run at base* | 

These findings and the escapes below are things to scan, not verdicts: in the [pilot](#what-weve-measured), most of them were true but harmless.

| Finding | Severity | 
|---|---|
| Test focused ( `it.only` ,`describe.only` ,`fit` ,`fdescribe` , RSpec`focus: true` , Dart`solo: true` ) | ✖️ FAIL | 
| Test skipped ( `it.skip` ,`xit` ,`@pytest.mark.skip` ,`t.Skip()` ,`#[ignore]` ,`@Disabled` ,`[Ignore]` ,`markTestSkipped` ,`XCTSkip` ,`skip:` …) |  | 
| Tests removed (net count; lines added in another changed test file count as moved, cancelling at most as many removals as were added there; not reported when the same file also adds a test definition and its assertions did not drop on net, i.e. tests merged; Rust inline tests are never treated as merged) |  | 
| Assertions removed (net count, including `self.assertEqual` style; lines added in another changed test file count as moved, cancelling at most as many removals as were added there) |  | 
| Snapshot changed or deleted |  | 
| Test or coverage config changed |  | 
| Nothing proves the change (code changed, and no test changed, no new or changed test that ran was found in the changed test files, or none fails without the change) |  | 

New lines that silence the type checker, linter or coverage:

| Language | Skip / focus markers | Escapes flagged | 
|---|---|---|
| TypeScript / JavaScript | `.only` ,`fit` ,`fdescribe` (FAIL);`.skip` ,`.todo` ,`xit` ,`xdescribe` | `as any` ,`: any` ,`@ts-ignore` ,`@ts-nocheck` ,`@ts-expect-error` ,`eslint-disable` ,`istanbul` /`c8` /`v8 ignore` | 
| Python | `@pytest.mark.skip` /`skipif` /`xfail` ,`@unittest.skip` ,`pytest.skip()` ,`self.skipTest()` | `# type: ignore` ,`# noqa` ,`# pragma: no cover` | 
| Go | `t.Skip()` ,`t.SkipNow()` ,`t.Skipf()` | `//nolint` ,`//lint:ignore` ,`//go:build ignore` ,`unsafe.*` | 
| Rust | `#[ignore]` | `#[allow(…)]` ,`#[expect(…)]` ,`unsafe` blocks | 
| Java | `@Disabled` ,`@Ignore` | `@SuppressWarnings` ,`@SuppressFBWarnings` ,`NOPMD` ,`CHECKSTYLE:OFF` | 
| Kotlin | `@Disabled` ,`@Ignore` | `@Suppress` ,`@SuppressLint` ,`ktlint-disable` | 
| C# | `[Ignore]` ,`[Explicit]` ,`Skip =` ,`Assert.Ignore()` | `#pragma warning disable` ,`SuppressMessage` ,`#nullable disable` ,`ExcludeFromCodeCoverage` | 
| Ruby | `focus: true` /`:focus` and`fit` (FAIL);`skip` ,`pending` ,`xit` | `rubocop:disable` ,`:nocov:` | 
| PHP | `markTestSkipped()` ,`markTestIncomplete()` , Pest`->skip()` /`->todo()` | `@phpstan-ignore` ,`@psalm-suppress` ,`phpcs:ignore` ,`@codeCoverageIgnore` ,`@` error suppression | 
| Dart | `solo: true` (FAIL);`skip: true` / a reason string,`@Skip` | `// ignore:` ,`// ignore_for_file:` ,`coverage:ignore-*` | 
| Swift | `XCTSkip` ,`XCTSkipIf` ,`.disabled(…)` | `swiftlint:disable` ,`@unchecked Sendable` | 

Checks look at real code and comments only, so a string or regex that merely *mentions* `it.only` doesn't trigger a false alarm. Removed tests and assertions are counted per language too (`@Test`, `func Test…`, `#[test]`, `assert_eq!`, `Assert.*`, …).

A ranked list of the files that deserve your attention: auth and permissions, database migrations, CI pipelines, dependencies, secrets and env files, infrastructure, agent config (`.claude/`, `AGENTS.md`, …), files with findings, and very large changes.

`trustbut verify --json` prints an **agent receipt**: an open, versioned JSON format for evidence about a code change. Any tool can produce one and any tool can read one, and trustbut is just the reference implementation. The [spec](https://github.com/isAlmogK/trustbut/blob/main/spec/agent-receipt-v1.md) defines a small required core (`specVersion`, `producer`, `subject`, `verdict`, `findings`), optional standard sections (` tests`, `proofs`, `risks`), 21 core finding ids with fixed meanings, namespaced ids like `acme/secret-leak` for everything else, and `x-` keys for tool-specific data.

Check any receipt, from trustbut or anything else, against the spec:

``` bash
$ trustbut verify --json | trustbut validate
valid
```

The schema is [`spec/agent-receipt-v1.schema.json`](https://github.com/isAlmogK/trustbut/blob/main/spec/agent-receipt-v1.schema.json), and examples are in [`spec/examples/`](https://github.com/isAlmogK/trustbut/blob/main/spec/examples). Building a tool that emits receipts? Open an issue and I'll link it here.

The trustbut GitHub Action adds a receipt to your pull requests: a sticky comment (updated in place on each push), the receipt in the job summary, and a failed job on FAIL.

```
# .github/workflows/trustbut.yml
name: trustbut
on: pull_request
permissions:
  contents: read
  pull-requests: write   # for the sticky comment
jobs:
  receipt:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0   # trustbut needs history to find the merge-base
      # add your language's setup and dependency install here (setup-node + npm ci, setup-go, ...)
      - uses: isAlmogK/trustbut@v0
```

The action installs Node 22 if the runner has none or one older than 22; that Node stays on the PATH for later steps in the job. Add your own language's `setup-*` step and dependency install before it, and a `.trustbut.json` in the repo if your tests need setup.

**Inputs** (all optional)

| Input | Default | Description | 
|---|---|---|
| `base` | the PR base branch, else the repo default branch | Base branch to compare against | 
| `strict` | `false` | Fail on WARN as well as FAIL | 
| `config` | none | Path to a trustbut config file | 
| `root` | none | Package folder to run tests in, relative to the repo root | 
| `working-directory` | `.` | Directory to run trustbut from | 
| `version` | the version matching the action's ref | npm version of trustbut to run, or `local` to build and run the action's own checkout | 
| `node-version` | none | Force a Node version (by default Node 22 is installed only if the runner has no Node 22 or newer) | 
| `comment` | `true` | Post or update the sticky PR comment | 
| `comment-key` | `default` | Tell several comments on one PR apart, for example one per monorepo package | 
| `github-token` | `${{ github.token }}` | Token for the comment (needs `pull-requests: write` ) | 
| `comment-author` | `github-actions[bot]` | Login whose comment is updated in place; set it when `github-token` is a PAT or GitHub App token | 

**Outputs**

| Output | Description | 
|---|---|
| `verdict` | `pass` ,`warn` ,`fail` or`error` | 
| `exit-code` | trustbut's exit code: `0` ,`1` for FAIL (or WARN with`strict` ),`2` for an error | 
| `receipt-path` | Path to the Markdown receipt | 
| `comment-url` | URL of the sticky comment (empty if none was posted) | 

**Pull requests from forks** get the receipt in the job summary only: GitHub gives them a read-only token, so no comment is posted. Always use the `pull_request` event, **never `pull_request_target`**: trustbut runs the PR's tests, and `pull_request_target` would run them with a write token and your secrets (the action prints a warning if you do).

``` php
flowchart LR
    A[base..head] --> B[git diff]
    B --> C[Classify files<br/>test · source · snapshot · config]
    C --> D[Static checks<br/>integrity · escapes]
    C --> E[Proof engine<br/>clean rerun at head<br/>new tests at base]
    C --> F[Risk ranking]
    D --> G[🧾 Receipt<br/>PASS · WARN · FAIL]
    E --> G
    F --> G
```

- **Throwaway worktrees.** Both runs happen in temporary`git worktree` s that are always removed afterwards. Your working directory is never touched. Dependencies (`node_modules` ,`.venv` ) are linked from your checkout instead of reinstalled; other toolchains (Go, Cargo, Maven, …) use their usual caches.
- **Runner and language registries.** Each runner is a small adapter that turns a test command into per-test results, and each language declares how to read its comments, strings and test markers. Adding either is one file.
- **Deterministic.** The same commits give the same receipt. There's no model, no sampling and no network call in the core.

## **Does it send my code anywhere or use an LLM?**

No. trustbut runs entirely on your machine with git and your own test runner. There is no AI model, no telemetry and no account.

## **Which agents does it work with?**

All of them. trustbut verifies the *result* (your git history and your tests), so it doesn't matter whether Claude Code, Codex, Cursor, Gemini CLI, OpenCode, Copilot or a human wrote the change. Agents with a turn-end hook (Claude Code, Codex, Gemini CLI, OpenCode) can also run trustbut automatically: see [Use it with your agent](#use-it-with-your-agent). Deeper integrations that read agent transcripts are on the roadmap.

## **Is it a replacement for code review?**

No. It's evidence to have *before* a review: did the tests really run, does any new or changed test prove the change, and what got silenced. The "Review these first" list points at the files that deserve attention. It doesn't judge whether the change is correct or well designed.

## **Do agents actually cheat on tests?**

Rarely, in merged pull requests. On 201 real PRs, no agent PR made a passing test fail, and 97% of agents' new or changed tests failed without the change, against 88% for people's (see [What we've measured](#what-weve-measured)). Lab reports show models do game tests during agentic work, which is what the hook checks before the agent says "done"; how often that catches something isn't measured yet.

## **Isn't running the tests twice slow?**

The base run only executes the new and changed test files, not the whole suite. Use `--quick` to also limit the head run to those files.

## **Is it safe to run?**

trustbut runs your repository's tests, which means running your code, the same trust as running `npm test` or `pytest`. It does this in temporary worktrees and never modifies your checkout. In CI, use `pull_request` (not `pull_request_target`), as in the example above.

With an agent hook, keep in mind that the check runs **outside the agent's sandbox**: at the end of each turn your tests, including ones the agent just wrote, and the `command`/` setup` in `.trustbut.json` run with your permissions and network access. Where that matters (an untrusted repo, an agent you run sandboxed), turn the hook off with `"hook": { "enabled": false }` in `.trustbut.json` or `TRUSTBUT_SKIP=1`, and run `trustbut verify` yourself in a sandbox.

## **Why "trustbut"?**

From "trust, but verify." Agents are genuinely useful, so keep trusting them, but check the evidence before you merge. The command reads as the idea: `trustbut verify`.

- **v0.2 (done):** every language, a runner registry and`.trustbut.json` ,`--working`
- **v0.3 (done, released as 0.3.x on npm):** the GitHub Action with a sticky PR comment, hooks that stop an agent from saying "done" on a red receipt (Claude Code, Codex, Gemini CLI, OpenCode), and the npm release
- **v0.4 (done):** the receipt format as an open spec ([agent receipts v1](https://github.com/isAlmogK/trustbut/blob/main/spec/agent-receipt-v1.md) ) with stable finding ids, and`trustbut validate`
- **0.5 to 0.7 (released):**`test-regressed` confirmed by a rerun, less noise from the pilot, and`change-not-proven` as the headline finding
- **Agent test-honesty index (study in progress):** measuring trustbut on real agent and human pull requests in[trustbut-index](https://github.com/isAlmogK/trustbut-index) ; first results are in[What we've measured](#what-weve-measured)
- **Later:** transcripts and claim checks, mutation testing on changed lines,`trustbut compare` , a local web UI

See the full [roadmap](https://github.com/isAlmogK/trustbut/blob/main/docs/ROADMAP.md) and the [design spec](https://github.com/isAlmogK/trustbut/blob/main/docs/SPEC.md).

Contributions are welcome, especially **real agent mistakes the receipt should have caught**. Open an issue with a minimal repro; it becomes a test scenario before it gets a fix.

```
npm ci
npm test           # unit and end-to-end tests (real git repos, real test runs)
npm run typecheck
npm run build
npm run receipt   # trustbut's receipt for your branch; run before opening a PR
```

Every pull request gets a trustbut receipt in the CI job summary, and pull requests from this repository also get it as a sticky comment (fork PRs get the job summary only, since GitHub gives them a read-only token). Run `npm run receipt` locally first so there are no surprises.

trustbut is built with AI coding agents and verified by itself: every pull request gets a trustbut receipt in CI (a sticky comment for branches in this repo; the job summary for forks). The rules agents follow in this repo are in [AGENTS.md](https://github.com/isAlmogK/trustbut/blob/main/AGENTS.md).

trustbut is free and open source, and I build it in my own time. If it saves you or your team from a bad merge, [sponsoring me on GitHub](https://github.com/sponsors/isAlmogK) keeps it maintained and moving down the [roadmap](https://github.com/isAlmogK/trustbut/blob/main/docs/ROADMAP.md).

I'm Almog Koren, a software engineer in Tallinn with more than 15 years of shipping production software. I founded and built [DogBase](https://www.dogbase.co), a bootstrapped SaaS for working-dog teams that won the main pitch competition at Latitude59 2026, and before that I was co-founder and CTO of Platfarm, which was acquired. I took both companies through Techstars. These days I mostly build AI agents and the systems around them, and trustbut came out of that work: I needed to know what my agents had actually done.

**I'm open to work:** full-time roles, fractional CTO work and AI agent projects. My CV and contact details are at [almog.io](https://almog.io).

Built by **Almog Koren** · [almog.io](https://almog.io) · [@isAlmogK](https://github.com/isAlmogK). If trustbut saves you from a bad merge, a ⭐ helps others find it, and [sponsoring](https://github.com/sponsors/isAlmogK) helps keep it going.