{"slug": "show-hn-ai-codebase-analyser-and-auto-fixer", "title": "Show HN: AI codebase analyser and auto-fixer", "summary": "Getdebug CLI 0.4.0 adds Python AI-app regex prefilters for five categories (prompt-injection, unsafe-role-merge, pii-in-prompt, unbounded-stream, unsafe-tool-output), running in milliseconds with no LLM call. In benchmark tests against Bandit, Semgrep, and vulnhuntr on 10 paired vulnerable/safe Python AI-app fixtures, getdebug achieved 83% precision and 100% recall, while Bandit and Semgrep each scored 50% precision and 20% recall; vulnhuntr selected zero files. On the real-world repo simonw/llm, getdebug produced 1,261 findings with high signal, compared to Bandit's 1,261 and Semgrep's 1,261.", "body_md": "getdebug CLI 0.4.0 ships Python AI-app regex prefilters alongside the JS/TS ones that landed in 0.3.0. Five categories: prompt-injection, unsafe-role-merge, pii-in-prompt, unbounded-stream, unsafe-tool-output. All deterministic. No LLM call. The default `getdebug analyze .`\n\non a Python repo runs in milliseconds, costs zero, and now covers Python AI-app anti-patterns alongside the secrets + dependency-CVE passes it always ran.\n\nThe interesting part isn't the patterns themselves — they're straightforward regex on familiar SDK idioms (`messages=[{\"role\": \"system\"...}]`\n\n, `stream=True`\n\n, `subprocess.run(tool_call.input.command)`\n\n). The interesting part is the benchmark we ran while developing them, against the three other tools anyone would compare us to.\n\n## The four tools, the honest map\n\n**Bandit** (PyCQA) is the Python-OSS standard security linter. Hand-written rules. Free, fast, no LLM. Python only.\n\n**Semgrep** is multi-language SAST with community rule packs. Hand-written rules. Free, fast, no LLM. Same product shape as getdebug.\n\n**vulnhuntr** (Protect AI, open source) is the stated category leader for AI-app static analysis. LLM-driven, Python-only, entry-point-detection based.\n\n**getdebug** is what we ship: pattern-based regex prefilters in JS/TS + Python now, plus an optional local-LLM SAST pass via Ollama (free, on-device) and a hosted LLM SAST pass via Claude (paid). The Python additions in 0.4.0 are the regex layer specifically.\n\n## Test 1 — paired vulnerable/safe fixtures (10 files)\n\nWe wrote 5 vulnerable + 5 safe Python AI-app fixtures, one pair per category. Same shape as our existing JS/TS corpus. All four tools ran on the same set.\n\n```\nTool        TP  FP  FN   Precision  Recall\ngetdebug     5   1   0    83%        100%\nbandit       1   1   4    50%        20%\nsemgrep      1   1   4    50%        20%\nvulnhuntr    —   —   —    (0 files selected; see below)\n```\n\nBandit and Semgrep both fire on the `unsafe-tool-output`\n\nfixture via their generic `subprocess.run(shell=True)`\n\nrules — that's a true positive on the vulnerable variant. But they also fire on the **safe** variant of the same fixture, the one where the model output is mapped through an allowlist before reaching the shell:\n\n```\n# Safe pattern — Bandit + Semgrep both flag this as a FP\nALLOWED = {\"hosts\": \"cat /etc/hosts\", \"uptime\": \"uptime\"}\ndef handle(tool_call):\n    cmd = ALLOWED.get(tool_call.input.tag)\n    if not cmd: return \"rejected\"\n    return subprocess.run(cmd, shell=True, capture_output=True).stdout\n```\n\nNeither tool knows that `cmd`\n\ncame from a static dict, not from the model. They see `shell=True`\n\nand fire. getdebug's regex requires the `tool_call.input.X`\n\n/ `block.input.X`\n\nreference to appear in the sink arg, which narrows the false-positive surface the generic SAST tools don't. Honesty update: the `scanPyShellToolArg`\n\ndetector added in 0.5.5 now fires a `CRITICAL`\n\non this same safe fixture — it sees `tool_call.input.tag`\n\nreaching a `shell=True`\n\nsink and can't tell the value was routed through the static allowlist first. Re-running the corpus on 0.5.5 scores getdebug at 5 TP / 1 FP / 0 FN (83% precision) on this set, not the clean sweep the original 0.4.0 numbers showed. Tightening this allowlist case is tracked.\n\nBoth tools miss the other four behavioural categories entirely — pii-in-prompt, unsafe-role-merge, prompt-injection, unbounded-stream. They aren't designed for them; the rule packs don't contain patterns for `{\"role\": \"system\", \"content\": f\"...{'$'}{name}...\"}`\n\n. That's our addition.\n\n## Test 2 — the real-world signal/noise check\n\nSynthetic fixtures lock the behaviour in. The real test for a security scanner is what it does on actual code. We ran all three (working) tools against [simonw/llm](https://github.com/simonw/llm) — Simon Willison's clean, well-maintained CLI for talking to LLMs, 49 Python files (commit 0d593ea; raw tool outputs in bench/realworld/).\n\n```\nTool        Total findings    Signal\nbandit      1,261            1,228 are 'assert_used' (pytest);\n                              zero AI-app coverage\nsemgrep     3                3 generic-SAST hits;\n                              zero AI-app coverage\ngetdebug    7                6 AI-app findings (1 prompt-injection,\n                              5 unbounded-stream) + 1 borderline role-merge\n```\n\nBandit's 1,261 findings on this 49-file codebase is *almost entirely noise*: 1,228 of them are `assert_used`\n\nwarnings on pytest assertions. This is a long-standing Bandit complaint — the default config flags every `assert`\n\nas a security smell because asserts disappear under `python -O`\n\n. For a production app where you opt-out of -O optimisation (which is almost everyone), this is pure noise.\n\nSemgrep's 3 findings are real but generic: an `exec()`\n\nusage flagged in `cli.py`\n\n(intentional for the plugin system), a missing-integrity attribute on a static HTML asset, and a non-literal import. None of these are AI-app specific.\n\nSix of getdebug's seven findings are AI-app categorized: one prompt-injection (the CLI's template feature concatenating two user-supplied strings) and five unbounded-stream hits in the OpenAI plugin (each `stream=True`\n\nwith no `with`\n\nblock or timeout in scope). Both arguable as TPs depending on threat model — this is a single-user CLI, so the prompt-injection between two CLI-user inputs is a stretch, and the library delegates stream-management to its consumer. The seventh, a role-merge on `{\"role\": role}`\n\n, is a borderline false positive — the role echoes the API's own streaming response, not request input — the kind of thing you'd triage away. Everything is flagged `HIGH`\n\n/ `MEDIUM`\n\n, not `CRITICAL`\n\n, with context to triage. The CLI suppresses findings with a `.getdebug-ignore`\n\nfile (inline `// getdebug:ignore`\n\nannotations are a hosted-scan feature).\n\n## About vulnhuntr\n\nvulnhuntr is the stated category leader for LLM-driven AI-app static analysis, so we re-checked the current release. Two findings, and we want to be fair about both:\n\n**It runs.** vulnhuntr 1.2.2 installs and starts cleanly on Python 3.11 — the older`ModuleNotFoundError`\n\nis gone. Like any LLM-driven tool it needs a provider (an API key, or a local Ollama via`--llm ollama`\n\n).**But it sees nothing here.** Its file-selection heuristic targets “network-exposed” entry points, and simonw/llm is a CLI, not a web app — so vulnhuntr selects**zero files** to analyse and surfaces nothing on this repo. Confirmed with its own`--dry-run`\n\n(0 files, $0.00, no key needed).\n\nSo vulnhuntr isn't broken — it's scoped to web-app entry points, which makes it a no-op on a CLI library like this one. Different tool, different target: getdebug's AI-app prefilters run on any Python that touches an LLM, web app or not. That's the gap we're shipping into.\n\n## What this means for you\n\nIf you ship a Python app that calls an LLM — chat wrapper, agent framework, tool-calling backend, batch summariser — you should run *all three*:\n\n`bandit -r .`\n\nfor general Python security hygiene (turn off`assert_used`\n\nvia`.bandit`\n\nconfig first).`semgrep --config auto .`\n\nfor cross-language SAST coverage.`npx @getdebug/cli@0.4.0 analyze .`\n\nfor the AI-app behavioural patterns the other two don't target.\n\nThey're complementary. None of them subsume the others. The first two catch general SAST and Python hygiene; getdebug catches the “serialised the whole user object into the LLM prompt” class of bugs that you can't hand-write a sustainable rule for in generic SAST without painful ergonomics.\n\nReproduce every number on this page at [getdebug.dev/bench](/bench). Corpus, methodology, and harness are open — [CodeSecBench](https://github.com/getdebug-ai/codesecbench) on GitHub.\n\nAnd as always: if you see a result you can't reproduce, or a pattern getdebug should catch but doesn't, the corpus is open — PRs welcome, harness is reproducible, methodology is documented. That's the whole point of doing the benchmark in public.\n\ntry it\n\nnpx @getdebug/cli@0.4.0 analyze .\n\nOr via Homebrew: `brew install getdebug-ai/tap/getdebug`\n\n— Fafa Agbetsise / Founder, getdebug.dev", "url": "https://wpnews.pro/news/show-hn-ai-codebase-analyser-and-auto-fixer", "canonical_source": "https://www.getdebug.dev/blog/python-ai-app-prefilters", "published_at": "2026-07-25 09:23:31+00:00", "updated_at": "2026-07-25 09:52:33.114234+00:00", "lang": "en", "topics": ["ai-safety", "developer-tools", "ai-tools"], "entities": ["getdebug", "Bandit", "Semgrep", "vulnhuntr", "Protect AI", "Simon Willison", "PyCQA"], "alternates": {"html": "https://wpnews.pro/news/show-hn-ai-codebase-analyser-and-auto-fixer", "markdown": "https://wpnews.pro/news/show-hn-ai-codebase-analyser-and-auto-fixer.md", "text": "https://wpnews.pro/news/show-hn-ai-codebase-analyser-and-auto-fixer.txt", "jsonld": "https://wpnews.pro/news/show-hn-ai-codebase-analyser-and-auto-fixer.jsonld"}}