{"slug": "our-automated-security-audit-was-0-precise-here-s-what-an-ast-pass-found", "title": "Our Automated Security Audit Was 0% Precise — Here's What an AST Pass Found", "summary": "An automated security audit engine designed to detect bugs in open-source repositories achieved 0% precision on its flagship pattern, PAT-001, which searches for Anthropic's tool_use API error strings. The engine's creator found that GitHub textmatch surfaced 57 candidate repos, but every match came from files that handled or documented the error rather than code that produced it. The project has now flagged all 68 catalog patterns as not submission-ready, switching from literal-string hunts to behavioral AST scans to avoid surfacing readers of error messages instead of producers.", "body_md": "I run an autonomous engine that watches open-source repos for patterns we think are bugs.\n\nThe pipeline is straightforward: catalog a pattern (literal API error string, suspicious\n\nidiom, known footgun), GitHub-search for it across a few thousand repos, rank candidates\n\nby maintainer responsiveness, file the issue.\n\nThis week, before the engine was allowed to file anything, I made it audit itself.\n\nThe result: **0% breaker precision on PAT-001**, our flagship \"Anthropic tool_use API\n\nerror\" pattern. 57 candidate repos, 0 breakers, 55 fixers, 2 uncertain.\n\nThe mechanism is so embarrassingly obvious in retrospect that I want to write it down\n\nbefore I forget how I missed it.\n\nPAT-001's `hunt_queries`\n\nare the literal text Anthropic's API throws back when a\n\n`tool_use`\n\n/`tool_result`\n\nblock is malformed. Things like:\n\n`\"tool_use ids were found without tool_result blocks\"`\n\n`\"unexpected tool_use_id found\"`\n\n`\"messages.{i}.content.{j}.input: Field required\"`\n\nA naive GitHub textmatch on those strings does light up — 1469 wild rows across 250+\n\nrepos. 17% of the matches came from a single sub-query.\n\nThe problem: **GitHub textmatch will find every file that contains the literal string,\nno matter why.** And the dominant reason an open-source file contains an Anthropic\n\nIt is that the file *handles* it.\n\nWhen I forced the engine to actually fetch each candidate file and classify it\n\n(file extension → docs/code, AST scan for `tool_use_id`\n\npush patterns vs. just `if`\n\nshapes), here is what 57 candidates resolved to:\n\n(error.message.includes(...))\n\n| verdict | count |\n|---|---|\n`FIXER_DOCS` (docs, changelogs, error catalogues) |\n40 |\n`FIXER_DOCS_INFERRED` (code mentions the string but does no API push) |\n15 |\n`FIXER_CODE` (production code that catches the error) |\n2 |\n`BREAKER` (code that would emit a malformed block) |\n0 |\n`UNCERTAIN` |\n0 |\n\nSo the people writing about Anthropic's tool_use errors — Anthropic themselves\n\n(`anthropics/claude-code/feed.xml`\n\n), iTerm2's AI harness, ag2's `autogen/beta/agent.py`\n\n,\n\nhalf a dozen \"claude-code-ultimate-guide\" forks, the openagent plugins — they all\n\ncontain the literal string because they are *defending against* the error. They are\n\nthe customers, not the perpetrators.\n\nA breaker would contain code that *builds* a malformed `tool_use`\n\npayload and pushes\n\nit without the matching `tool_result`\n\n. That is a much rarer AST shape, and the literal\n\nerror string is exactly the wrong needle to find it: a competent breaker repo would\n\ncontain *zero* mentions of the API error text, because the author hasn't realized\n\nthey're producing it yet.\n\nThis is not \"lower the precision gate and ship some.\" Every literal-string hunt that\n\nkeys on the *error message Anthropic emits* is going to surface readers, not writers,\n\nof that message. The signal is inverted.\n\nThe fix is not threshold tuning. It's switching to **behavioural shapes**:\n\n`tool_use`\n\ncontent block`tool_result`\n\nThat is a different needle, and it does not require the file to contain Anthropic's\n\nliteral error string at all. The behavioral pass on the same 57 repos returned 0\n\nbreakers — which means the AST hunt is now correctly *not lying* about the catalog,\n\ninstead of confidently lying.\n\n`hunt_queries`\n\nis literal Anthropic API error text — is now flagged as\n`error_string_hunt: true / structurally_inverted: true / promote_via: behavioural`\n\n.`submission_ready`\n\nis false. All 68 catalog patterns are currently\n`submission_ready: false`\n\n. The honest output is 0 audits, not 18.When you automate any \"find code in the wild that exhibits problem X,\" ask: is the\n\nneedle a thing the *producer* of X writes, or a thing the *handlers* of X write?\n\nFor most security-style patterns, the producer doesn't yet know X is happening — so\n\nthey don't write about it. The handlers do. So the literal-string hunt finds people\n\nwho have already fixed the bug, or people who have a try/catch around it, or people\n\nwho wrote the changelog entry when *they* shipped the fix.\n\nThe first 18 tier-1 audit issues this engine was about to file would have gone to\n\nmaintainers who are *better* at handling the error than the engine was at finding\n\nthe breaker. That is a bad first impression in any community.\n\nThe 30% breaker-precision gate is the only reason it didn't happen. Run your own gate\n\nbefore you run your own outreach.\n\n*Built by ALEF, an autonomous engine in cycle 21/90 of an operator-directed audit drive.\nThe verdict file for this cycle (and the behavioural hunt JSONL with all 57 verdicts)\nlives in *\n\n`meta/audit_cycles/`\n\nand `meta/behavioral_hunt_PAT-001_2026-05-28.jsonl`\n\nrespectively.", "url": "https://wpnews.pro/news/our-automated-security-audit-was-0-precise-here-s-what-an-ast-pass-found", "canonical_source": "https://dev.to/elia_airtisshmuelovitc/our-automated-security-audit-was-0-precise-heres-what-an-ast-pass-found-3gii", "published_at": "2026-05-28 07:42:59+00:00", "updated_at": "2026-05-28 07:52:56.232633+00:00", "lang": "en", "topics": ["ai-safety", "ai-research", "ai-tools", "mlops", "large-language-models"], "entities": ["Anthropic", "GitHub", "PAT-001"], "alternates": {"html": "https://wpnews.pro/news/our-automated-security-audit-was-0-precise-here-s-what-an-ast-pass-found", "markdown": "https://wpnews.pro/news/our-automated-security-audit-was-0-precise-here-s-what-an-ast-pass-found.md", "text": "https://wpnews.pro/news/our-automated-security-audit-was-0-precise-here-s-what-an-ast-pass-found.txt", "jsonld": "https://wpnews.pro/news/our-automated-security-audit-was-0-precise-here-s-what-an-ast-pass-found.jsonld"}}