Trustbut checks if an AI agent's new tests fail without its change

Your agent says it's done and the tests pass. Does anything actually prove the change?

🧾 Verified by trustbut: pull requests to this repo carry its own trustbut receipt.

Languages · Why · What we've measured · Install · Usage · Use it with your agent · Config · What it checks · Receipt format · CI · How it works · FAQ · Roadmap · Sponsor · About me

trustbut is a local command-line tool that produces a receipt for a code change, whether an AI agent or a person wrote it. It reruns your tests in a clean checkout, runs every new or changed test against the code from before the change to see whether it fails there (if it passes either way, it doesn't test the change), and flags skipped tests, removed assertions and silenced type checks. Then it gives you a PASS / WARN / FAIL verdict. Wired into your agent, it can also stop the agent from finishing while the receipt is FAIL, so it keeps fixing until the evidence holds up (details).

It works with any agent (Claude Code, Codex, Cursor, Gemini CLI, OpenCode, Copilot) and 11 languages (JavaScript/TypeScript, Python, Go, Rust, Java, Kotlin, C#, Ruby, PHP, Dart/Flutter and Swift) because it only needs git and your test runner. No LLM, no server, no account. Your code never leaves your machine.

$ npx trustbut verify

## 🧾 Receipt: WARN

`3f2a91c` → `8be04d1` · runner: vitest

**Tests (clean checkout):** 41 passed · 0 failed · 1 skipped
**Proof:** 1 new/changed test proves the change ✅
**New/changed tests:** 1 prove the change · 1 pass without it

### Findings
- ⚠ **Test skipped** · `test/auth.test.ts:88`: it.skip('handles clock skew', () => {
- ⚠ **cast to any** · `src/auth/session.ts:41`: const claims = decoded as any;

### Review these first
1. `src/auth/session.ts`: auth/permissions, has findings
2. `package.json`: dependencies

When the suite fails at head, the receipt also lists the failing tests. If package.json declares dependencies that aren't installed in your node_modules (a stale install), a deps-not-installed warning names them, because both runs link and use your installed dependencies. See the finding ids in the spec's finding catalog.

The output is Markdown, so the same receipt reads cleanly in your terminal and renders properly in a pull request or a CI job summary.

Language	Test frameworks	Tamper and escape checks	Verified in CI against the real toolchain
JavaScript / TypeScript	Vitest, Jest (incl. Expo / React Native), Mocha, node:test	✅	✅
Python	pytest	✅	✅
Go	go test	✅	✅
Rust	cargo-nextest, cargo test	✅	✅
Java	Maven, Gradle	✅	✅
Kotlin	Maven, Gradle	✅	◐ same runners as Java; Kotlin end-to-end test coming
C# / .NET	dotnet test (xUnit verified; NUnit and MSTest via the same TRX results)	✅	✅
Ruby	RSpec	✅	✅
PHP	PHPUnit	✅	✅
Dart / Flutter	dart test ,flutter test	✅	✅
Swift	Swift Package Manager (XCTest, Swift Testing)	✅	✅
Anything else	any command that writes JUnit XML	—	✅

trustbut detects the right runner automatically, including in monorepo subfolders. Setup steps (such as starting a database emulator), extra arguments and environment variables go in a .trustbut.json. Detection rules and the exact commands are in Supported languages and test runners.

AI agents now write a large share of the code I ship. Writing it is no longer the slow part. Checking it is.

When an agent says "done, all tests pass", that's the agent reporting on its own work. Reading the diff doesn't show what actually ran, or whether the new tests would fail if the change were wrong. I wanted a check from outside the agent that answers those two questions with evidence, not with another model's opinion.

The data says I'm not alone:

• In the 2026 Stack Overflow survey ,48% of developers trust AI output when they can easily validate it, but only16% trust it for most tasks. The gap between those numbers is the problem trustbut works on.
• In 2025, 66% of developers named "almost right, but not quite" as their top frustration with AI tools (InfoWorld on the 2025 survey ).
• Armin Ronacher put it well in The Final Bottleneck : generating code is now cheaper than reviewing it, and the human stays accountable.

The labs have also reported their models gaming tests in agentic coding:

• Anthropic's Claude 3.7 Sonnet system card reports the model special-casing tests, and sometimes modifying them, to make them pass in agentic coding environments like Claude Code. TheClaude 4 system card reports this much reduced, but still present.
• OpenAI's research on monitoring reasoning models caught a model exiting early or skipping unit tests to look green.
• METR found frontier models patching the grading and scoring code of their evaluations, andImpossibleBench measures how often models exploit tests on tasks that can't be solved honestly.

That's why trustbut also flags skipped tests, removed assertions and silenced checks. But in my own measurement of 201 real pull requests, merged agent work rarely showed these behaviours (see What we've measured). The question that holds up is the plainer one: does anything prove this change?

AI code reviewers read the diff and comment, but they don't rerun anything. A vendor checking its own agent is a weak audit, and most teams use more than one agent. So trustbut is independent, deterministic and local: it doesn't ask an AI whether the code looks right. It runs things and reports what it found.

I'm measuring whether trustbut is useful, in the open, in trustbut-index. It runs trustbut on real merged pull requests from established open-source projects: PRs written by agents (Copilot, Codex, Claude Code, Cursor, Devin, Jules) and, as a control, PRs written by people in the same projects. These are pilot results, not the final study.

trustbut 0.7.0 on 201 pull requests:

	Agent PRs	Human PRs
trustbut could run the tests	53 of 103	56 of 98
A test that passed before the change fails after it¹	0 of 53	0 of 56
New or changed tests that fail without the change, so they test it	275 of 283 (97%)	294 of 333 (88%)
PRs with any finding	12 of 53	15 of 56
PRs where nothing proves the change ( change-not-proven )	6 of 53	7 of 56

¹ 18 of the 109 runs had a failing test with no result from before the change, so this count is a lower bound.

What this says so far:

• Agents didn't break suites or fake their tests in these PRs. Their new tests were at least as likely as people's to test the change. Merged PRs have already been through CI and review, though, so this says little about what an agent does before it says "done". That's where theagent hook works, and it isn't measured yet.
• Most findings are true but don't need action. In the 0.6.0 pilot, 71 findings were labeled: 5 were worth a reviewer acting on (7%), 57 were true but harmless, and 9 were wrong. All 5 useful ones said that nothing proved the change. 0.7.0 fixed the causes of the 9 wrong findings and made that one its headline,change-not-proven . The labels are drafts by Claude that no person has confirmed yet, and the labeler could see the code, so they weren't fully blind.
• trustbut ran on about half of the PRs. Of the 92 it couldn't run, 78 failed to install the project's dependencies or toolchain in the study's runner, 13 had no supported test runner, and 1 result is missing.

Not measured yet: whether change-not-proven findings are worth acting on, and whether the agent hook leads to better agent work.

On my own repos (DogBase), receipts found a fix commit with no test, 7 new tests that passed with or without their change, two new eslint-disable lines, and a dependency missing from the local install, which became the deps-not-installed warning.

Have a run of your own? Open an issue with what the receipt found, or missed.

Requirements: Node.js 22 or newer, git, and a repository with tests in one of the supported languages and test runners, or any runner that can write JUnit XML. The toolchain of your project (Go, Cargo, Maven, .NET, Composer, …) must be installed, as it is when you run the tests yourself.

Run without installing (recommended):

npx trustbut verify

Install globally:

npm i -g trustbut
trustbut verify

From source:

git clone https://github.com/isAlmogK/trustbut.git
cd trustbut
npm ci
npm link

To have your agent verify its own work after every turn, see Use it with your agent.

Run it in your repository after an agent finishes:

trustbut verify            # the committed work on your branch
trustbut verify --working  # uncommitted work too (tracked edits plus untracked files)

By default trustbut compares your branch with the point where it split off main (or master), the same way a GitHub pull request shows its diff.

trustbut verify --base develop
trustbut verify --base HEAD~3

trustbut verify --working

trustbut verify --quick

trustbut verify --root services/api
trustbut verify --config ~/configs/myapp.trustbut.json

trustbut verify --strict

trustbut verify --json > receipt.json

Option	Description	Default
-b, --base <ref>	Base branch or commit, compared from its merge-base with head	origin/main ,main ,origin/master ormaster
--head <ref>	The commit to verify (not with --working )	HEAD
--working	Verify uncommitted work: tracked edits plus untracked, non-ignored files, snapshotted without touching your index	off
--runner <name>	Force a test runner, any name from the table below	detected automatically
--root <dir>	Package folder to run the tests in, relative to the repo root (overrides root in the config)	.
--config <path>	Use this config file instead of .trustbut.json at head; it may live outside the repo	.trustbut.json
--quick	At head, rerun only the new and changed test files	off (full suite)
--strict	Exit with 1 on WARN as well as FAIL	off
--json	Print the receipt as JSON: an agent receipt v1	off
--receipt-json <path>	Also write the JSON receipt to this file (relative to the current directory; parent folders are created)	off
--output-tail	JSON receipts: add the end of each failed run's output (last 50 lines, at most 4,000 characters) as outputTail underx-trustbut . Tests can print secrets	off
--receipt-md <path>	Also write the Markdown receipt to this file	off

Verdict	Meaning	Exit code
✅ PASS	Tests are green from a clean checkout, the new tests prove the change, and nothing suspicious was found	0
WARN	Something deserves a human look: a change no test proves, a skipped test, a silenced check	0 (1 with--strict )
✖️ FAIL	Tests fail from a clean checkout, a changed test fails, or a test was focused ( .only )	1
—	Usage or runtime error, e.g. an unknown ref or option, or a --receipt-json /--receipt-md file that couldn't be written	2

Tip

By default trustbut only looks at committed changes. If you have uncommitted work, the receipt says so. Commit first, or pass --working to include it.

trustbut can run itself at the end of every agent turn, so a FAIL receipt never reaches you as "done".

Note

The commands below assume trustbut is installed globally (npm i -g trustbut).

What happens

1. When the agent finishes a turn, trustbut quickly verifies its work: your uncommitted changes plus the commits on your branch. By default it reruns only the new and changed test files (the whole suite when none changed).
2. FAIL: the agent gets the receipt (failing tests with their messages, focused tests, and so on) and keeps fixing. This repeats up tohook.maxRetries times in a row (default 3), then the agent is allowed to stop and you see the FAIL.
3. WARN or PASS: you get a one-line note and the agent finishes.
4. trustbut never blocks on its own errors: a timeout, a crash or a missing tool lets the agent finish with a "couldn't verify" note.
5. A tree already verified isn't rerun. A FAIL is sent back once per tree: if the agent stops again without changing anything, or you start a new turn with nothing changed, it is allowed to finish and you see the FAIL with the receipt path.

After each completed check, the full Markdown receipt is at .git/trustbut/last-receipt.md (per worktree; inside .git, so it is never committed).

Things to know:

• The hooks run on macOS and Linux (POSIX) for now; Windows isn't supported yet.
• Use either the Claude Code plugin or trustbut init claude , not both: with both, every turn is checked twice.
• The hook runs your tests outside the agent's sandbox; see Is it safe to run? .

Install the plugin from this repository's marketplace:

/plugin marketplace add isAlmogK/trustbut
/plugin install trustbut@trustbut

Or write the hook into your project without the plugin: trustbut init claude (use one or the other, not both). The plugin uses the trustbut on your PATH when it is recent enough, and otherwise runs npx trustbut@<version>. Both run the hook in exec form ( args), which needs a recent Claude Code (2.1 or later).

When trustbut sends a FAIL back, Claude Code labels it "Stop hook error occurred". This is expected: it is how Claude Code shows a blocking Stop hook, and the receipt is still delivered to the agent.

trustbut init codex

Codex asks you to approve new hooks: open /hooks and approve trustbut verify (needs Codex 0.129 or later). Project hooks load only in trusted projects.

trustbut init gemini

Project hooks run only in trusted folders, so trust the folder in Gemini CLI if the hook doesn't fire. Needs Gemini CLI 0.35 or later.

trustbut init opencode

This writes a plugin for the OpenCode V1 plugin API (opencode-ai 1.x; V2 support is a follow-up). OpenCode can't block "done", so on a FAIL the receipt is sent back to the agent as a follow-up prompt instead.

trustbut init <claude|codex|gemini|opencode> copies a small runner script next to the agent's config and, for Claude Code, Codex and Gemini CLI, merges one hook entry into it, leaving your other hooks alone. For OpenCode it writes a plugin file instead. It is safe to rerun: it updates its own entry in place.

Option	What it does
--dry-run	Print what would change and write nothing
--user	Set up your user-level config ( ~/.claude ,~/.codex ,~/.gemini ,~/.config/opencode ) instead of the project

Project files are written at the git root, and agents read project settings from the directory they were started in, so start the agent from the repo root. A settings file that contains comments is refused (exit 2) with the snippet to add by hand.

Set TRUSTBUT_SKIP=1 in the environment to skip the check for a session (for example TRUSTBUT_SKIP=1 claude). To change how it behaves, add a hook object to .trustbut.json. It is read as committed at HEAD, so an uncommitted edit by the agent can't switch it off.

{ "hook": { "mode": "full", "maxRetries": 2 } }

Key	Type	Default	Description
enabled	boolean	true	false lets every stop through without verifying
mode	"quick" or"full"	"quick"	quick reruns only the new and changed test files at head;full reruns the whole suite
maxRetries	whole number, 0 to 100	3	How many times in a row a FAIL is sent back to the agent. 0 reports a FAIL without ever blocking
timeoutSec	number above 0	540	Time budget for one check, kept under the agents' 600 s hook timeout. When it runs out the agent is allowed to finish
base	ref name	origin's default branch, then origin/main ,main ,origin/master ,master , then the branch's upstream, elseHEAD	What to compare against

The hook protocols, exit codes and environment variables are in the spec.

trustbut picks the runner by looking at the package folder (the repo root, or root / --root) as committed at head (with --working, your working tree). If several match, the most specific one wins; force one with --runner <name> or "runner" in the config.

Runner	Language	Detected by	Runs	Notes
vitest	JS / TS	vitest dependency, orvitest.config.* /vitest.workspace.*	vitest run --reporter=junit	Wins over Jest when both are present
jest	JS / TS	jest dependency or config,jest.config.* , or ajest-expo /react-native preset	jest --ci --json --runTestsByPath <files>	Works for Expo and React Native apps
mocha	JS / TS	mocha dependency or.mocharc.*	mocha --reporter json
node-test	JS / TS	scripts.test inpackage.json usesnode --test	node --test	Uses the built-in node:test runner
pytest	Python	pytest.ini ,conftest.py ,tox.ini , or[tool.pytest inpyproject.toml	python -m pytest -q --junitxml	Uses the checkout's .venv /venv if present
go	Go	go.mod	go test -json -count=1 ./...	Add "args": ["-p", "1"] for packages that share a database or emulator
cargo	Rust	Cargo.toml	cargo nextest run if installed, elsecargo test --no-fail-fast	Also checks #[cfg(test)] modules in source files. Proofs come from integration tests undertests/ : inline#[cfg(test)] tests run in the suite but give no fail-before / pass-after proof, since they live in the source file being changed
maven	Java / Kotlin	pom.xml (themvnw wrapper is preferred)	mvn -B test (Surefire reports)
gradle	Java / Kotlin	build.gradle(.kts) orsettings.gradle(.kts) (thegradlew wrapper is preferred)	gradle test --continue
dotnet	C#	a .sln ,.slnx ,.csproj or.fsproj in the root	dotnet test (TRX results)	With several projects it prefers the solution file
rspec	Ruby	.rspec , orspec/ plusrspec in theGemfile	bundle exec rspec --format json	Plain rspec when there is no Gemfile
phpunit	PHP	phpunit.xml(.dist) , orphpunit/phpunit incomposer.json	vendor/bin/phpunit --log-junit	Runs composer install in the temporary checkout, sincevendor/ isn't tracked
dart	Dart	pubspec.yaml with atest dev dependency	dart test --reporter json	Runs dart pub get first when.dart_tool is missing
flutter	Dart / Flutter	pubspec.yaml with the Flutter SDK	flutter test --machine
swift	Swift	Package.swift	swift test --parallel --verbose --xunit-output	Swift Package Manager projects
command	anything	never detected; chosen with command in the config	your command	See below

Each runner's own report (JUnit XML, JSON events or TRX, depending on the tool) is turned into the same per-test results, so each language gets the same fail-before / pass-after proof.

If your tests run through something not listed, point trustbut at a command that writes JUnit XML:

{
  "command": "./run-tests.sh {files}",
  "junit": "test/reports/junit.xml"
}

{files} is replaced with the quoted changed test files (and with nothing for a full run). Without junit, trustbut reads the command's stdout if it is JUnit XML. Setting command selects the command runner automatically.

Everything is optional; most projects need no config at all. Put .trustbut.json in the repo root. trustbut reads it as committed at head (so the version under review is the one used), then falls back to the working-tree file. --config <file> uses a different file instead, which can live outside the repo. Unknown keys and wrong types are errors (exit code 2) that list the allowed keys; a $schema key is ignored.

Key	Type	Default	Description
root	string	"."	Package folder relative to the repo root. Runners are detected and run there, and only changed test files under it are rerun (changed tests elsewhere are listed as not run). Must stay inside the repo and exist at head (otherwise exit code 2 ).--root overrides it.
runner	string	detected	Force a runner from the table above. --runner overrides it.
command	string	none	The command runner: shell command to run.{files} becomes the quoted test files (nothing for a full run).
junit	string	none	The command runner: JUnit XML file the command writes, relative toroot .
setup	string	none	Shell command started in the background before each test run (for example an emulator or database), and always killed afterwards. If it exits non-zero before the tests start, the run stops with exit code2 .
setupReadyPattern	regex string	none	Tests start as soon as setup prints a line matching this (stdout or stderr).
setupWaitSec	number, 0 to 2,000,000	5	How long to wait for setup ; withsetupReadyPattern it is the maximum wait.
env	object of strings	none	Extra environment variables for every test run and for setup .
args	string[]	none	Extra arguments added to the runner's command: before the test files (or packages, selectors) for most runners; for Maven, Gradle and .NET at the end; for swift after the--filter options; for thecommand runner appended (shell-quoted) at the very end of the command.
timeoutSec	number, up to 2,000,000	900	Time limit for each test run; the receipt says when a run timed out.

1. A Go backend with a Firebase emulator, in a subfolder

{
  "root": "backend",
  "args": ["-p", "1"],
  "setup": "firebase emulators:start --only firestore,auth",
  "setupReadyPattern": "All emulators ready",
  "setupWaitSec": 90,
  "env": { "FIRESTORE_EMULATOR_HOST": "127.0.0.1:8080" }
}

-p 1 runs packages one at a time so they don't fight over the emulator. The emulator is started before the head run and again before the base run, and always shut down.

2. An Expo / React Native app with Jest

{
  "runner": "jest",
  "args": ["--silent"]
}

Usually you don't need any config: jest-expo and react-native presets are detected. Dependencies are linked from your checkout, so there's no reinstall.

3. Vitest in a functions/ subfolder

{ "root": "functions" }

Or skip the file and run trustbut verify --root functions.

4. Any other tool, through the command runner

{
  "command": "./scripts/run-tests.sh {files}",
  "junit": "build/test-results/junit.xml",
  "timeoutSec": 1800
}

Warning

command and setup are shell commands that run as part of verification, from the config at head, just like a changed test script in package.json would. Only verify code you'd be willing to run.

Your test suite runs in a fresh, temporary git worktree at head, not in your working directory and not by trusting the agent's own run. A red suite is a FAIL.

This is the core idea. Every new or changed test also runs against the base commit:

Before the change	After the change	Result
fails	passes	✅ Proves the change
passes	passes	Passes without the change : the test doesn't check the new behaviour. Listed in the receipt; the receipt warns only when no new or changed test proves the change
—	fails	✖️ Fails at head
—	skipped	Skipped
couldn't run (e.g. imports code that didn't exist yet)	passes	✅ Counted, but marked as couldn't run at base

These findings and the escapes below are things to scan, not verdicts: in the pilot, most of them were true but harmless.

Finding	Severity
Test focused ( it.only ,describe.only ,fit ,fdescribe , RSpecfocus: true , Dartsolo: true )	✖️ FAIL
Test skipped ( it.skip ,xit ,@pytest.mark.skip ,t.Skip() ,#[ignore] ,@Disabled ,[Ignore] ,markTestSkipped ,XCTSkip ,skip: …)
Tests removed (net count; lines added in another changed test file count as moved, cancelling at most as many removals as were added there; not reported when the same file also adds a test definition and its assertions did not drop on net, i.e. tests merged; Rust inline tests are never treated as merged)
Assertions removed (net count, including self.assertEqual style; lines added in another changed test file count as moved, cancelling at most as many removals as were added there)
Snapshot changed or deleted
Test or coverage config changed
Nothing proves the change (code changed, and no test changed, no new or changed test that ran was found in the changed test files, or none fails without the change)

New lines that silence the type checker, linter or coverage:

Language	Skip / focus markers	Escapes flagged
TypeScript / JavaScript	.only ,fit ,fdescribe (FAIL);.skip ,.todo ,xit ,xdescribe	as any ,: any ,@ts-ignore ,@ts-nocheck ,@ts-expect-error ,eslint-disable ,istanbul /c8 /v8 ignore
Python	@pytest.mark.skip /skipif /xfail ,@unittest.skip ,pytest.skip() ,self.skipTest()	# type: ignore ,# noqa ,# pragma: no cover
Go	t.Skip() ,t.SkipNow() ,t.Skipf()	//nolint ,//lint:ignore ,//go:build ignore ,unsafe.*
Rust	#[ignore]	#[allow(…)] ,#[expect(…)] ,unsafe blocks
Java	@Disabled ,@Ignore	@SuppressWarnings ,@SuppressFBWarnings ,NOPMD ,CHECKSTYLE:OFF
Kotlin	@Disabled ,@Ignore	@Suppress ,@SuppressLint ,ktlint-disable
C#	[Ignore] ,[Explicit] ,Skip = ,Assert.Ignore()	#pragma warning disable ,SuppressMessage ,#nullable disable ,ExcludeFromCodeCoverage
Ruby	focus: true /:focus andfit (FAIL);skip ,pending ,xit	rubocop:disable ,:nocov:
PHP	markTestSkipped() ,markTestIncomplete() , Pest->skip() /->todo()	@phpstan-ignore ,@psalm-suppress ,phpcs:ignore ,@codeCoverageIgnore ,@ error suppression
Dart	solo: true (FAIL);skip: true / a reason string,@Skip	// ignore: ,// ignore_for_file: ,coverage:ignore-*
Swift	XCTSkip ,XCTSkipIf ,.disabled(…)	swiftlint:disable ,@unchecked Sendable

Checks look at real code and comments only, so a string or regex that merely mentions it.only doesn't trigger a false alarm. Removed tests and assertions are counted per language too (@Test, func Test…, #[test], assert_eq!, Assert.*, …).

A ranked list of the files that deserve your attention: auth and permissions, database migrations, CI pipelines, dependencies, secrets and env files, infrastructure, agent config (.claude/, AGENTS.md, …), files with findings, and very large changes.

trustbut verify --json prints an agent receipt: an open, versioned JSON format for evidence about a code change. Any tool can produce one and any tool can read one, and trustbut is just the reference implementation. The spec defines a small required core (specVersion, producer, subject, verdict, findings), optional standard sections ( tests, proofs, risks), 21 core finding ids with fixed meanings, namespaced ids like acme/secret-leak for everything else, and x- keys for tool-specific data.

Check any receipt, from trustbut or anything else, against the spec:

$ trustbut verify --json | trustbut validate
valid

The schema is spec/agent-receipt-v1.schema.json, and examples are in spec/examples/. Building a tool that emits receipts? Open an issue and I'll link it here.

The trustbut GitHub Action adds a receipt to your pull requests: a sticky comment (updated in place on each push), the receipt in the job summary, and a failed job on FAIL.

name: trustbut
on: pull_request
permissions:
  contents: read
  pull-requests: write   # for the sticky comment
jobs:
  receipt:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0   # trustbut needs history to find the merge-base
      - uses: isAlmogK/trustbut@v0

The action installs Node 22 if the runner has none or one older than 22; that Node stays on the PATH for later steps in the job. Add your own language's setup-* step and dependency install before it, and a .trustbut.json in the repo if your tests need setup.

Inputs (all optional)

Input	Default	Description
base	the PR base branch, else the repo default branch	Base branch to compare against
strict	false	Fail on WARN as well as FAIL
config	none	Path to a trustbut config file
root	none	Package folder to run tests in, relative to the repo root
working-directory	.	Directory to run trustbut from
version	the version matching the action's ref	npm version of trustbut to run, or local to build and run the action's own checkout
node-version	none	Force a Node version (by default Node 22 is installed only if the runner has no Node 22 or newer)
comment	true	Post or update the sticky PR comment
comment-key	default	Tell several comments on one PR apart, for example one per monorepo package
github-token	${{ github.token }}	Token for the comment (needs pull-requests: write )
comment-author	github-actions[bot]	Login whose comment is updated in place; set it when github-token is a PAT or GitHub App token

Outputs

Output	Description
verdict	pass ,warn ,fail orerror
exit-code	trustbut's exit code: 0 ,1 for FAIL (or WARN withstrict ),2 for an error
receipt-path	Path to the Markdown receipt
comment-url	URL of the sticky comment (empty if none was posted)

Pull requests from forks get the receipt in the job summary only: GitHub gives them a read-only token, so no comment is posted. Always use the pull_request event, never pull_request_target: trustbut runs the PR's tests, and pull_request_target would run them with a write token and your secrets (the action prints a warning if you do).

flowchart LR
    A[base..head] --> B[git diff]
    B --> C[Classify files<br/>test · source · snapshot · config]
    C --> D[Static checks<br/>integrity · escapes]
    C --> E[Proof engine<br/>clean rerun at head<br/>new tests at base]
    C --> F[Risk ranking]
    D --> G[🧾 Receipt<br/>PASS · WARN · FAIL]
    E --> G
    F --> G

• Throwaway worktrees. Both runs happen in temporarygit worktree s that are always removed afterwards. Your working directory is never touched. Dependencies (node_modules ,.venv ) are linked from your checkout instead of reinstalled; other toolchains (Go, Cargo, Maven, …) use their usual caches.
• Runner and language registries. Each runner is a small adapter that turns a test command into per-test results, and each language declares how to read its comments, strings and test markers. Adding either is one file.
• Deterministic. The same commits give the same receipt. There's no model, no sampling and no network call in the core.

Does it send my code anywhere or use an LLM? #

No. trustbut runs entirely on your machine with git and your own test runner. There is no AI model, no telemetry and no account.

Which agents does it work with? #

All of them. trustbut verifies the result (your git history and your tests), so it doesn't matter whether Claude Code, Codex, Cursor, Gemini CLI, OpenCode, Copilot or a human wrote the change. Agents with a turn-end hook (Claude Code, Codex, Gemini CLI, OpenCode) can also run trustbut automatically: see Use it with your agent. Deeper integrations that read agent transcripts are on the roadmap.

Is it a replacement for code review? #

No. It's evidence to have before a review: did the tests really run, does any new or changed test prove the change, and what got silenced. The "Review these first" list points at the files that deserve attention. It doesn't judge whether the change is correct or well designed.

Do agents actually cheat on tests? #

Rarely, in merged pull requests. On 201 real PRs, no agent PR made a passing test fail, and 97% of agents' new or changed tests failed without the change, against 88% for people's (see What we've measured). Lab reports show models do game tests during agentic work, which is what the hook checks before the agent says "done"; how often that catches something isn't measured yet.

Isn't running the tests twice slow? #

The base run only executes the new and changed test files, not the whole suite. Use --quick to also limit the head run to those files.

Is it safe to run? #

trustbut runs your repository's tests, which means running your code, the same trust as running npm test or pytest. It does this in temporary worktrees and never modifies your checkout. In CI, use pull_request (not pull_request_target), as in the example above.

With an agent hook, keep in mind that the check runs outside the agent's sandbox: at the end of each turn your tests, including ones the agent just wrote, and the command/ setup in .trustbut.json run with your permissions and network access. Where that matters (an untrusted repo, an agent you run sandboxed), turn the hook off with "hook": { "enabled": false } in .trustbut.json or TRUSTBUT_SKIP=1, and run trustbut verify yourself in a sandbox.

Why "trustbut"? #

From "trust, but verify." Agents are genuinely useful, so keep trusting them, but check the evidence before you merge. The command reads as the idea: trustbut verify.

• v0.2 (done): every language, a runner registry and.trustbut.json ,--working
• v0.3 (done, released as 0.3.x on npm): the GitHub Action with a sticky PR comment, hooks that stop an agent from saying "done" on a red receipt (Claude Code, Codex, Gemini CLI, OpenCode), and the npm release
• v0.4 (done): the receipt format as an open spec (agent receipts v1 ) with stable finding ids, andtrustbut validate
• 0.5 to 0.7 (released):test-regressed confirmed by a rerun, less noise from the pilot, andchange-not-proven as the headline finding
• Agent test-honesty index (study in progress): measuring trustbut on real agent and human pull requests intrustbut-index ; first results are inWhat we've measured
• Later: transcripts and claim checks, mutation testing on changed lines,trustbut compare , a local web UI

See the full roadmap and the design spec.

Contributions are welcome, especially real agent mistakes the receipt should have caught. Open an issue with a minimal repro; it becomes a test scenario before it gets a fix.

npm ci
npm test           # unit and end-to-end tests (real git repos, real test runs)
npm run typecheck
npm run build
npm run receipt   # trustbut's receipt for your branch; run before opening a PR

Every pull request gets a trustbut receipt in the CI job summary, and pull requests from this repository also get it as a sticky comment (fork PRs get the job summary only, since GitHub gives them a read-only token). Run npm run receipt locally first so there are no surprises.

trustbut is built with AI coding agents and verified by itself: every pull request gets a trustbut receipt in CI (a sticky comment for branches in this repo; the job summary for forks). The rules agents follow in this repo are in AGENTS.md.

trustbut is free and open source, and I build it in my own time. If it saves you or your team from a bad merge, sponsoring me on GitHub keeps it maintained and moving down the roadmap.

I'm Almog Koren, a software engineer in Tallinn with more than 15 years of shipping production software. I founded and built DogBase, a bootstrapped SaaS for working-dog teams that won the main pitch competition at Latitude59 2026, and before that I was co-founder and CTO of Platfarm, which was acquired. I took both companies through Techstars. These days I mostly build AI agents and the systems around them, and trustbut came out of that work: I needed to know what my agents had actually done.

I'm open to work: full-time roles, fractional CTO work and AI agent projects. My CV and contact details are at almog.io.

Built by Almog Koren · almog.io · @isAlmogK. If trustbut saves you from a bad merge, a ⭐ helps others find it, and sponsoring helps keep it going.

source & further reading

github.com — original article Current memory tools can pass a poisoned run on. Procedural memory can stop it Show HN: ThinkBoard – Summon an Obsidian canvas above your desktop with a hotkey Show HN: Pi-cowboy – a Herdr native agent orchestrator for Pi