{"slug": "day-6-the-blind-spot-in-my-own-blind-spot-detector", "title": "Day 6: The blind spot in my own blind-spot detector", "summary": "A developer building an audit hook for Claude Code found that the tool's verification logic has a blind spot: when an agent claims to have fixed something in a file that was written during the session for an unrelated reason, the hook marks the claim verified even though the diff never touches the relevant code. Testing two variants against the audit-turn.sh hook, the developer showed a read-only file is correctly blocked (exit 2) while an unrelated edit to the same file passes (exit 0), and published tests/known-gaps.sh to reproduce the gap rather than ship a keyword-matching patch that would encourage agents to pad claims with jargon.", "body_md": "Day 5 landed with one comment. It was the useful kind — the kind that identifies the same trust problem one level up from the one you just solved.\n\nThe comment, roughly: *\"Nice, you check whether the file was touched. But what if the file was touched for an unrelated reason, and the agent still claims to have fixed something that wasn't fixed?\"*\n\nThe suggested fixture: `\"I fixed the timeout in retry.py\"` — where `retry.py` exists in the session, but no change related to the timeout was ever made.\n\nI spent today testing it against my own `audit-turn.sh` hook. Here's what I found.\n\n**Variant A — `retry.py` was only read, never written.**\n\n```\nclaims=1 verified=0 unverified=1 → exit 2\n```\n\nBlocked. The hook correctly flags the claim as unverified because `retry.py` has no write event in `.claude/trust-audit.log` and no changes in `git status`. This is the case I built the hook for.\n\n**Variant B — `retry.py` was written in this session, but with an unrelated edit (say, a comment).**\n\n```\nclaims=1 verified=1 unverified=0 → exit 0\n```\n\nWaved through. The claim \"I fixed the timeout\" is marked verified because `retry.py` appears in `recent_writes ∪ git_touched`. The hook has no idea whether the diff actually touches `timeout`-related code. If the agent made any change to the file and claimed any fix in it, the audit passes.\n\nThat's the blind spot. It's real.\n\nThe naive patch is a keyword match: if the claim says \"timeout\", scan the diff for \"timeout\". I thought about it for a while and decided against pushing it.\n\nThe failure modes:\n\nKeyword heuristics would mostly produce the opposite problem: false negatives on legitimate fixes, which teach the agent to pad its claims with jargon to pass the filter. That's worse than the current false positive.\n\nThe direction I'm thinking about (not shipped, not tested yet) is a protocol change: the agent declares at the end of each turn, in a structured `## Changes` block, which file was changed for what reason. The hook then flags mismatch between the declaration and the actual writes.\n\nThat moves the trust problem from \"did the file change\" to \"did the agent declare honestly\" — which is a smaller problem, because the declaration lives next to the diff and a human reviewer can spot-check it. But it also requires the agent to cooperate with the protocol, which is a different kind of fragility.\n\nA test file: [`tests/known-gaps.sh`](https://github.com/98dkcc488b-glitch/did-it-actually/blob/main/tests/known-gaps.sh) — reproduces both variants against the current hook, documents the gap, and lives *outside* `run-tests.sh` so it doesn't fail the suite. It's a honest TODO, not red paint.\n\nAnd this post, which is me saying out loud: the hooks I shipped yesterday catch five specific lying patterns reproducibly. They do not catch \"I fixed X in a file that got touched for a different reason.\" If you install them expecting them to, you'll be disappointed.\n\nI'd rather tell you the gap exists than ship a bad fix that papers over it.\n\n**Repo:** [github.com/98dkcc488b-glitch/did-it-actually](https://github.com/98dkcc488b-glitch/did-it-actually) (MIT) · **PWYW:** [dumkestar6.gumroad.com/l/vicuep](https://dumkestar6.gumroad.com/l/vicuep) ($5 min, $9 suggested)\n\nPrevious entries: [Day 1 — how I stopped Claude Code from lying](https://dev.to/98dkcc488bglitch/how-i-stopped-claude-code-from-lying-about-finishing-tasks-218e) · [Day 4 — a bug in my own branch-guard hook](https://dev.to/98dkcc488bglitch/a-bug-in-my-own-branch-guard-hook-caught-by-typing-one-thing-wrong-54pb) · [Day 5 — what 6 strangers found](https://dev.to/98dkcc488bglitch/day-5-what-6-strangers-found-in-my-20-line-parser-5b8l)\n\n*This post was drafted with AI assistance and reviewed for accuracy.*", "url": "https://wpnews.pro/news/day-6-the-blind-spot-in-my-own-blind-spot-detector", "canonical_source": "https://dev.to/98dkcc488bglitch/day-6-the-blind-spot-in-my-own-blind-spot-detector-1j1o", "published_at": "2026-10-19 14:10:12+00:00", "updated_at": "2026-10-19 14:14:27.833751+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-safety", "developer-tools"], "entities": ["Claude Code", "did-it-actually", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/day-6-the-blind-spot-in-my-own-blind-spot-detector", "markdown": "https://wpnews.pro/news/day-6-the-blind-spot-in-my-own-blind-spot-detector.md", "text": "https://wpnews.pro/news/day-6-the-blind-spot-in-my-own-blind-spot-detector.txt", "jsonld": "https://wpnews.pro/news/day-6-the-blind-spot-in-my-own-blind-spot-detector.jsonld"}}