Show HN: Snitch – Deterministic prose claim verifier for coding agents. (OSS)

Snitch, an open-source deterministic prose claim verifier for AI coding agents, has been released. It monitors transcripts from tools like Cursor and Claude Code to flag false claims made by AI agents, such as assertions that tests pass or code was committed, by cross-referencing tool calls, filesystem state, and git history.

Snitch watches your AI agent so you don't have to. Install install · Help train Snitch coming soon help-train-snitch-coming-soon · Roadmap roadmap Snitch is a deterministic prose claim verifier for AI coding agents. It watches transcripts from Cursor https://cursor.com , Claude Code https://claude.com/claude-code , Codex https://github.com/openai/codex , Pi https://pi.dev , and OpenCode https://opencode.ai , extracts high-confidence claims from assistant text "all tests pass", "I committed this" , and flags claims contradicted by evidence: tool calls including subagent merges , tool output, filesystem, git, session lookback 3 prior turns , and same-turn consistency. brew tap fristovic/snitch brew install snitch snitch start Snitch Bar opens in the menu bar and starts the claim verifier automatically. Latest release: curl -fsSL https://raw.githubusercontent.com/fristovic/snitch/main/scripts/install.sh | bash From a cloned repo: ./scripts/install.sh After install, open Snitch Bar: snitch start - Downloads or builds snitch CLI and Snitch Bar.app includes snitchd inside the app - Installs CLI to ~/.local/bin - Installs Snitch Bar.app to ~/.local/share/snitch/ - Registers a LaunchAgent to open Snitch Bar at login Open Snitch Bar once — it lives in the menu bar with no Dock icon: snitch start From the Snitch menu: | Action | What it does | |---|---| Snitching… / Paused / Offline | Current detection status | Start Snitching / Stop Snitching | Turn claim verification on or off | Latest: … | Preview of the most recent flagged claim type + short quote | View Details… | Open Terminal with full details snitch log --run <id | History ▸ Open Dashboard… | Open the interactive TUI snitch dashboard | Preferences… | Open ~/.snitch/config.yaml | Quit Snitch Bar | Stop the daemon and exit | When a false claim is caught, the menu bar icon alerts and Snitch Bar may show a Notification Center alert Snitch app icon . Click View Details… for the full verification breakdown, or History ▸ Open Dashboard… to browse history. snitch status is detection running? snitch dashboard browse runs and flagged claims interactively snitch log --run <id full detail for one agent turn snitch doctor install checklist Snitch stores every agent turn as a run with a verdict and claims . A false claim is a high-confidence prose claim inside a run that evidence contradicts. | View | Best for | What you see | |---|---|---| snitch log --run <id | One agent turn | Full breakdown — verdict, prompt, tool calls, every claim with evidence. | snitch dashboard | Browsing history | Interactive TUI — flip between runs and flagged claims, filter, search, live refresh. | Menu bar shortcuts: View Details… runs snitch log --run <id for the latest flagged claim. History ▸ Open Dashboard… runs snitch dashboard . snitch log --run abc12345 snitch dashboard | Item | Description | |---|---| Start / Stop Snitching | Pause or resume claim verification | | Alert icon | Flashes when a new false claim is caught | Latest: … | Disabled preview of the most recent flagged claim | View Details… | Open snitch log --run <id for the latest flagged claim | History ▸ Open Dashboard… | Open snitch dashboard in Terminal | Preferences… | Edit ~/.snitch/config.yaml | Quit Snitch Bar | Stop snitchd and exit | | Command | Description | |---|---| snitch start | Open Snitch Bar | snitch status | Detection status --detailed for per-harness stats | snitch log --run <id | Full verification detail for one run --trace , --json | snitch log --harness <name | List recent runs for one agent platform | snitch dashboard | Interactive TUI for runs and flagged claims --harness filter | snitch replay <path | Run any transcript through the pipeline offline — measure accuracy on your own sessions | snitch doctor | Debug install checklist per-harness | snitch uninstall | Remove daemon and binaries --purge for data | snitch config | View/set configuration | Snitch runs passively after install — it reads each enabled agent's local transcripts see Supported agents supported-agents ; Cursor's ~/.cursor/projects is watched by default. When Snitch Bar receives a failed or optionally warned run, it posts a macOS Notification Center alert attributed to Snitch Bar.app Snitch head icon . Configure in ~/.snitch/config.yaml : notifications: enabled: true on warn: false rate limit s: 5 The first notification triggers the macOS permission prompt for Snitch Bar. | Type | Example prose | Contradiction | |---|---|---| test pass | "all tests pass" | No test run, or test output shows failure | command ran | "I ran the command" | No shell tool call in the turn | command succeeded | "command ran successfully" | Shell exited with error | committed | "I committed" | No new commit since turn start | pushed | "I pushed" | No git push shell call | file created | "created foo.go" | No matching Write + file missing | file modified | "updated foo.go" | No matching Write / StrReplace + file missing | file deleted | "deleted foo.go" | No matching Delete / StrReplace + file still present | stub | "fully implemented" | Written file is a placeholder panic "TODO" , … | no action | action claims | Zero mutating tool calls in the turn | self contradiction | "won't modify X" | Tool call edits X in the same turn | count mismatch | "updated all 5 files" | File tool-call count ≠ 5 | negation violation | "did not touch tests" | test. file edited in the turn | tool write | Write tool call | File missing / empty after write | tool str replace | StrReplace tool call | Edit not reflected on disk | tool delete | Delete tool call | File still exists | tool shell | Shell tool call | Command evidence mismatch | tool read / tool glob / tool task | Read / Glob / Task | Tool effect vs disk / subagent evidence | Snitch persists each turn's full payload tool calls, git HEAD, file manifest in SQLite. When verifying recap/summary prose, it can credit evidence from up to three prior turns in the same session for: committed / pushed — git shell or HEAD delta in prior turns test pass / command — shell evidence in prior turns file created / file modified / file deleted — file tools + manifests in prior turns stub — placeholder bodies in files written this turn or prior turns Same-turn only: no action , self contradiction , count mismatch , and negation violation never use cross-turn lookback. Recap segments Summary , Summary , horizontal rules are tagged separately: inaccurate recap claims cap at WARN unless there is zero evidence across the current turn plus lookback. Snitch watches transcripts from five AI coding agents. Cursor is enabled by default; the others are opt-in. | Agent | Format | Location | Enable | |---|---|---|---| Cursor | JSONL | ~/.cursor/projects | on by default | Claude Code | JSONL | ~/.claude/projects | snitch config set platforms.claude.enabled true | Codex | JSONL | ~/.codex/sessions | snitch config set platforms.codex.enabled true | Pi | JSONL | ~/.pi/agent/sessions | snitch config set platforms.pi.enabled true | OpenCode | SQLite | ~/.local/share/opencode/opencode.db | snitch config set platforms.opencode.enabled true | After enabling a platform, restart Snitch snitch start . Each platform's claims, tool calls, and shell output are normalized to a common internal vocabulary, so the verification pipeline works identically across all five. The dashboard accepts a --harness filter to scope to one agent: snitch dashboard --harness claude . A community labeling flywheel — mark whether Snitch was right or wrong, report missed claims, and optionally share training examples — is coming soon . Labels stay local by default. When sharing is enabled dual opt-in: telemetry.enabled + share flag , a shared example may include: - the claim sentence full assistant sentence containing the match - a short surrounding context capped ±1–2 sentences - Snitch’s claimed → actual pair - metadata: claim type, harness, model, verdict, your label, and a hash for dedup Never shared: user prompts, full transcripts, source code, file paths, project paths, or shell dumps. 0.4.x this release : Claim-first UX flagged sentence → checked , tool types, UNUserNotificationCenter alerts. 0.3.x: Multi-harness ingestion Cursor + Claude Code + Codex + Pi + OpenCode , session lookback, Snitch Bar notifications with app icon. Coming soon: Community labeling and opt-in sync of claim sentences + short context to train a false-positive filter. Later: A locally-run false-positive classifier trained on community labels — reduces alert noise by filtering regex hits that aren't genuine claims. Snitchworks: A paid team layer — centralized dashboard, policy engine, premium semantic claim extraction. - Deterministic regex extraction only no LLM claim parsing — semantic extraction is a later goal - Lookback is limited to the current agent session 3 turns , not cross-session history - Subagent tool calls are merged by time window , not tool use id mapping - Consistency checks remain same-turn only - File manifests hash paths touched by tool calls at turn end; out-of-band disk changes may be missed By participating, you agree to follow the Code of Conduct /fristovic/snitch/blob/main/CODE OF CONDUCT.md .