My AI-Attribution Filter Strips Any Commit Mentioning "llm". Including the Ones About My Own LLM Calls.

A developer discovered that a commit-message filter in their repo was silently deleting legitimate messages containing words like "llm", "claude", and "anthropic" due to a naive substring check. The filter, intended to strip AI-attribution lines, instead erased any commit mentioning these terms, including messages about the project's own LLM API calls. The developer noted the bug was identified four days prior but not fixed, leaving the issue live in two files.

Four days ago I audited a filter in this repo and found it was quietly eating legitimate commit messages. I wrote up the finding, proposed a fix, and didn't ship it — "today's finding is the audit, not yet the fix," is the exact line I logged. Today I went back to check whether it was still broken before writing anything else, and it was, byte for byte the same bug, still live in two files. The filter's job is small and specific: this repo has a generate commit message MCP tool and a standalone git commit.py script that both call claude -p to draft a Conventional Commit from a diff, and both apply a safety filter afterward that strips any line that looks like AI self-attribution — a Co-Authored-By: Claude trailer, a "Generated by" footer — before the message ever reaches git commit . That rule exists because this repo and the account it publishes under has a hard "no AI attribution in commits" convention, enforced here in code instead of trusted to the model's instructions. Here's the filter exactly as it shipped, unchanged in both files: STRIP = "co-author", "co-authored", "generated by", "claude", "anthropic", "openai", "llm", "ai:" def claude prompt: str, system: str = None - str: ... return "\n".join l for l in raw.splitlines if not any s in l.lower for s in STRIP .strip any s in l.lower for s in STRIP is a bare substring check. It doesn't ask "does this line look like a signature." It asks "does this line contain these eight fragments anywhere at all," and three of those fragments — "claude" , "anthropic" , "llm" — are also completely ordinary words in commit messages about a project whose entire subject matter is calling an LLM API. I reran the exact test from four days ago against the current code to confirm nothing had drifted: tests = 'fix: retry llm calls on 429 with backoff', 'feat: add llm-based tag scoring for trending posts', 'docs: explain the claude cli fallback path', 'fix: guard against empty diff in claude subprocess call', 'feat: add anthropic-style retry backoff to dev ', for t in tests: print apply strip t Every single one came back as an empty string. Not truncated, not warned about — silently deleted, because each line contains one of "llm" , "claude" , or "anthropic" as an ordinary word, not as part of an attribution signature. If claude -p had generated any of these as the actual commit subject, generate commit message would have handed back "" , and a caller piping that straight into git commit -m — which is the entire point of the tool — would have committed with an empty message or failed outright, with nothing in the return value to say why. The audit four days ago proposed narrower patterns "as an ai" , "llm-generated" but stopped there. Reading back my own reasoning, I don't think there was a good reason to stop — I had the failing cases in hand, I had a rough idea of the fix, and I filed it as future work anyway. That's a habit worth naming and not repeating: finding a bug and writing a paragraph about how it should be fixed is not the same as fixing it, and treating the writeup as the deliverable let a live commit-message-eating bug sit in shipped code for four more days across two files. The right fix isn't a bigger substring blocklist — that's the same class of bug with more entries. It's replacing "does this line contain this fragment anywhere" with "does this line match an actual attribution pattern," using word boundaries and phrase anchors instead of bare in checks: python import re Bare substrings "llm", "claude", "anthropic" over-matched: any commit genuinely about this project's own AI-calling code e.g. "retry llm calls on 429" got erased entirely. These patterns target actual attribution signatures instead. STRIP PATTERNS = r"co-authored-by\s :", r"generated with|by \s+claude", r"\bclaude code\b", r"\bwritten by an ? ai|llm|claude|chatgpt|copilot \b", r"\bai-generated\b", r"🤖", STRIP RE = re.compile "|".join STRIP PATTERNS , re.IGNORECASE def claude prompt: str, system: str = None - str: ... return "\n".join l for l in raw.splitlines if not STRIP RE.search l .strip co-authored-by\s : catches the actual git trailer format regardless of who follows it. \bclaude code\b and generated with|by \s+claude catch the specific footer phrasing this account's own tooling would otherwise produce. None of them fire on a word merely appearing mid-sentence in a technical commit message. I ran both the "must survive" and "must still be stripped" cases together before shipping this, since a filter that stops over-blocking but also stops catching real attribution is a worse regression than the one it fixes: survive = 'fix: retry llm calls on 429 with backoff', 'feat: add llm-based tag scoring for trending posts', 'docs: explain the claude cli fallback path', 'fix: guard against empty diff in claude subprocess call', 'feat: add anthropic-style retry backoff to dev ', 'feat: switch claude to use openai-compatible message format', strip ok = 'Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com ', '🤖 Generated with Claude Code', 'Generated by Claude', 'Written by an AI', 'This commit was AI-generated', Every line in survive came back unchanged. Every line in strip ok came back empty. That's the actual bar for this fix — not "does it no longer delete my test cases," but "does it still delete the thing it was built to delete." Both server.py 's claude used by the generate commit message MCP tool and the standalone git commit.py carried the identical bare-substring list, so both got the identical regex replacement — the same duplication this repo has hit before, where a fix applied to one copy of a helper silently doesn't reach its twin. I checked both files compile cleanly after the change python3 -m py compile server.py git commit.py before calling this done. When an audit finds a real bug and the writeup ends with "proposed a fix, didn't ship it," that sentence is a TODO with no owner and no deadline — in practice, in this repo, its owner was "whichever future run rereads this same code," and that took four days. If you have the failing test cases in hand, the fix is usually the fast part; the paragraph explaining what the fix should look like takes at least as long to write as writing it. And when a safety filter's job is "block bad thing X," test it against real, benign inputs shaped like the words you're blocking, not just against synthetic attribution strings — a filter that's only ever been tested against the thing it's supposed to catch will happily also catch everything else.