{"slug": "show-hn-snitch-deterministic-prose-claim-verifier-for-coding-agents-oss", "title": "Show HN: Snitch – Deterministic prose claim verifier for coding agents. (OSS)", "summary": "Snitch, an open-source deterministic prose claim verifier for AI coding agents, has been released. It monitors transcripts from tools like Cursor and Claude Code to flag false claims made by AI agents, such as assertions that tests pass or code was committed, by cross-referencing tool calls, filesystem state, and git history.", "body_md": "**Snitch watches your AI agent so you don't have to.**\n\n[Install](#install) · [Help train Snitch (coming soon)](#help-train-snitch-coming-soon) · [Roadmap](#roadmap)\n\nSnitch is a deterministic prose claim verifier for AI coding agents. It watches transcripts from [Cursor](https://cursor.com), [Claude Code](https://claude.com/claude-code), [Codex](https://github.com/openai/codex), [Pi](https://pi.dev), and [OpenCode](https://opencode.ai), extracts high-confidence claims from assistant text (\"all tests pass\", \"I committed this\"), and flags claims contradicted by evidence: tool calls (including subagent merges), tool output, filesystem, git, session lookback (3 prior turns), and same-turn consistency.\n\n```\nbrew tap fristovic/snitch\nbrew install snitch\nsnitch start\n```\n\nSnitch Bar opens in the menu bar and starts the claim verifier automatically.\n\nLatest release:\n\n```\ncurl -fsSL https://raw.githubusercontent.com/fristovic/snitch/main/scripts/install.sh | bash\n```\n\nFrom a cloned repo:\n\n```\n./scripts/install.sh\n```\n\nAfter install, open Snitch Bar:\n\n```\nsnitch start\n```\n\n- Downloads or builds\n`snitch`\n\nCLI and**Snitch Bar.app**(includes`snitchd`\n\ninside the app) - Installs CLI to\n`~/.local/bin`\n\n- Installs\n**Snitch Bar.app** to`~/.local/share/snitch/`\n\n- Registers a LaunchAgent to open Snitch Bar at login\n\nOpen Snitch Bar once — it lives in the menu bar with no Dock icon:\n\n```\nsnitch start\n```\n\nFrom the Snitch menu:\n\n| Action | What it does |\n|---|---|\nSnitching… / Paused / Offline |\nCurrent detection status |\nStart Snitching / Stop Snitching |\nTurn claim verification on or off |\nLatest: … |\nPreview of the most recent flagged claim (type + short quote) |\nView Details… |\nOpen Terminal with full details (`snitch log --run <id>` ) |\nHistory ▸ Open Dashboard… |\nOpen the interactive TUI (`snitch dashboard` ) |\nPreferences… |\nOpen `~/.snitch/config.yaml` |\nQuit Snitch Bar |\nStop the daemon and exit |\n\nWhen a false claim is caught, the menu bar icon alerts and Snitch Bar may show a Notification Center alert (Snitch app icon). Click **View Details…** for the full verification breakdown, or **History ▸ Open Dashboard…** to browse history.\n\n```\nsnitch status             # is detection running?\nsnitch dashboard          # browse runs and flagged claims interactively\nsnitch log --run <id>     # full detail for one agent turn\nsnitch doctor             # install checklist\n```\n\nSnitch stores every agent **turn** as a **run** (with a verdict and claims). A **false claim** is a high-confidence prose claim inside a run that evidence contradicts.\n\n| View | Best for | What you see |\n|---|---|---|\n`snitch log --run <id>` |\nOne agent turn | Full breakdown — verdict, prompt, tool calls, every claim with evidence. |\n`snitch dashboard` |\nBrowsing history | Interactive TUI — flip between runs and flagged claims, filter, search, live refresh. |\n\n**Menu bar shortcuts:** **View Details…** runs `snitch log --run <id>`\n\nfor the latest flagged claim. **History ▸ Open Dashboard…** runs `snitch dashboard`\n\n.\n\n```\nsnitch log --run abc12345\nsnitch dashboard\n```\n\n| Item | Description |\n|---|---|\nStart / Stop Snitching |\nPause or resume claim verification |\n| Alert icon | Flashes when a new false claim is caught |\nLatest: … |\nDisabled preview of the most recent flagged claim |\nView Details… |\nOpen `snitch log --run <id>` for the latest flagged claim |\nHistory ▸ Open Dashboard… |\nOpen `snitch dashboard` in Terminal |\nPreferences… |\nEdit `~/.snitch/config.yaml` |\nQuit Snitch Bar |\nStop `snitchd` and exit |\n\n| Command | Description |\n|---|---|\n`snitch start` |\nOpen Snitch Bar |\n`snitch status` |\nDetection status (`--detailed` for per-harness stats) |\n`snitch log --run <id>` |\nFull verification detail for one run (`--trace` , `--json` ) |\n`snitch log --harness <name>` |\nList recent runs for one agent platform |\n`snitch dashboard` |\nInteractive TUI for runs and flagged claims (`--harness` filter) |\n`snitch replay <path>` |\nRun any transcript through the pipeline offline — measure accuracy on your own sessions |\n`snitch doctor` |\nDebug install checklist (per-harness) |\n`snitch uninstall` |\nRemove daemon and binaries (`--purge` for data) |\n`snitch config` |\nView/set configuration |\n\nSnitch runs passively after install — it reads each enabled agent's local transcripts (see [Supported agents](#supported-agents)); Cursor's `~/.cursor/projects`\n\nis watched by default.\n\nWhen Snitch Bar receives a failed (or optionally warned) run, it posts a macOS Notification Center alert attributed to **Snitch Bar.app** (Snitch head icon). Configure in `~/.snitch/config.yaml`\n\n:\n\n```\nnotifications:\n  enabled: true\n  on_warn: false\n  rate_limit_s: 5\n```\n\nThe first notification triggers the macOS permission prompt for Snitch Bar.\n\n| Type | Example prose | Contradiction |\n|---|---|---|\n`test_pass` |\n\"all tests pass\" | No test run, or test output shows failure |\n`command_ran` |\n\"I ran the command\" | No shell tool call in the turn |\n`command_succeeded` |\n\"command ran successfully\" | Shell exited with error |\n`committed` |\n\"I committed\" | No new commit since turn start |\n`pushed` |\n\"I pushed\" | No `git push` shell call |\n`file_created` |\n\"created foo.go\" | No matching `Write` + file missing |\n`file_modified` |\n\"updated foo.go\" | No matching `Write` /`StrReplace` + file missing |\n`file_deleted` |\n\"deleted foo.go\" | No matching `Delete` /`StrReplace` + file still present |\n`stub` |\n\"fully implemented\" | Written file is a placeholder (`panic(\"TODO\")` , …) |\n`no_action` |\naction claims | Zero mutating tool calls in the turn |\n`self_contradiction` |\n\"won't modify X\" | Tool call edits X in the same turn |\n`count_mismatch` |\n\"updated all 5 files\" | File tool-call count ≠ 5 |\n`negation_violation` |\n\"did not touch tests\" | `*_test.*` file edited in the turn |\n`tool_write` |\nWrite tool call | File missing / empty after write |\n`tool_str_replace` |\nStrReplace tool call | Edit not reflected on disk |\n`tool_delete` |\nDelete tool call | File still exists |\n`tool_shell` |\nShell tool call | Command evidence mismatch |\n`tool_read` / `tool_glob` / `tool_task` |\nRead / Glob / Task | Tool effect vs disk / subagent evidence |\n\nSnitch persists each turn's full payload (tool calls, git HEAD, file manifest) in SQLite. When verifying recap/summary prose, it can credit evidence from up to **three prior turns** in the same session for:\n\n`committed`\n\n/`pushed`\n\n— git shell or HEAD delta in prior turns`test_pass`\n\n/`command_*`\n\n— shell evidence in prior turns`file_created`\n\n/`file_modified`\n\n/`file_deleted`\n\n— file tools + manifests in prior turns`stub`\n\n— placeholder bodies in files written this turn or prior turns\n\n**Same-turn only:** `no_action`\n\n, `self_contradiction`\n\n, `count_mismatch`\n\n, and `negation_violation`\n\nnever use cross-turn lookback.\n\nRecap segments (`### Summary`\n\n, `## Summary`\n\n, horizontal rules) are tagged separately: inaccurate recap claims cap at WARN unless there is zero evidence across the current turn plus lookback.\n\nSnitch watches transcripts from five AI coding agents. Cursor is enabled by default; the others are opt-in.\n\n| Agent | Format | Location | Enable |\n|---|---|---|---|\nCursor |\nJSONL | `~/.cursor/projects` |\non by default |\nClaude Code |\nJSONL | `~/.claude/projects` |\n`snitch config set platforms.claude.enabled true` |\nCodex |\nJSONL | `~/.codex/sessions` |\n`snitch config set platforms.codex.enabled true` |\nPi |\nJSONL | `~/.pi/agent/sessions` |\n`snitch config set platforms.pi.enabled true` |\nOpenCode |\nSQLite | `~/.local/share/opencode/opencode.db` |\n`snitch config set platforms.opencode.enabled true` |\n\nAfter enabling a platform, restart Snitch (`snitch start`\n\n). Each platform's claims, tool calls, and shell output are normalized to a common internal vocabulary, so the verification pipeline works identically across all five.\n\nThe dashboard accepts a `--harness`\n\nfilter to scope to one agent: `snitch dashboard --harness claude`\n\n.\n\nA community labeling flywheel — mark whether Snitch was right or wrong, report missed claims, and optionally share training examples — is **coming soon**. Labels stay local by default.\n\nWhen sharing is enabled (dual opt-in: `telemetry.enabled`\n\n+ share flag), a shared example may include:\n\n- the\n**claim sentence**(full assistant sentence containing the match) - a\n**short surrounding context**(capped ±1–2 sentences) - Snitch’s\n**claimed → actual** pair - metadata: claim type, harness, model, verdict, your label, and a hash for dedup\n\n**Never shared:** user prompts, full transcripts, source code, file paths, project paths, or shell dumps.\n\n**0.4.x (this release):** Claim-first UX (flagged sentence → checked),`tool_`\n\n* types,`UNUserNotificationCenter`\n\nalerts.**0.3.x:** Multi-harness ingestion (Cursor + Claude Code + Codex + Pi + OpenCode), session lookback, Snitch Bar notifications with app icon.**Coming soon:** Community labeling and opt-in sync of claim sentences + short context to train a false-positive filter.**Later:** A locally-run false-positive classifier trained on community labels — reduces alert noise by filtering regex hits that aren't genuine claims.**Snitchworks:** A paid team layer — centralized dashboard, policy engine, premium semantic claim extraction.\n\n- Deterministic regex extraction only (no LLM claim parsing) — semantic extraction is a later goal\n- Lookback is limited to the current agent session (3 turns), not cross-session history\n- Subagent tool calls are merged by\n**time window**, not`tool_use_id`\n\nmapping - Consistency checks remain same-turn only\n- File manifests hash paths touched by tool calls at turn end; out-of-band disk changes may be missed\n\nBy participating, you agree to follow the [Code of Conduct](/fristovic/snitch/blob/main/CODE_OF_CONDUCT.md).", "url": "https://wpnews.pro/news/show-hn-snitch-deterministic-prose-claim-verifier-for-coding-agents-oss", "canonical_source": "https://github.com/fristovic/snitch", "published_at": "2026-07-10 13:44:08+00:00", "updated_at": "2026-07-10 14:06:01.239809+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-safety"], "entities": ["Snitch", "Cursor", "Claude Code", "Codex", "Pi", "OpenCode"], "alternates": {"html": "https://wpnews.pro/news/show-hn-snitch-deterministic-prose-claim-verifier-for-coding-agents-oss", "markdown": "https://wpnews.pro/news/show-hn-snitch-deterministic-prose-claim-verifier-for-coding-agents-oss.md", "text": "https://wpnews.pro/news/show-hn-snitch-deterministic-prose-claim-verifier-for-coding-agents-oss.txt", "jsonld": "https://wpnews.pro/news/show-hn-snitch-deterministic-prose-claim-verifier-for-coding-agents-oss.jsonld"}}