# Show HN: AI codebase analyser and auto-fixer > Source: > Published: 2026-07-25 09:23:31+00:00 getdebug CLI 0.4.0 ships Python AI-app regex prefilters alongside the JS/TS ones that landed in 0.3.0. Five categories: prompt-injection, unsafe-role-merge, pii-in-prompt, unbounded-stream, unsafe-tool-output. All deterministic. No LLM call. The default `getdebug analyze .` on a Python repo runs in milliseconds, costs zero, and now covers Python AI-app anti-patterns alongside the secrets + dependency-CVE passes it always ran. The interesting part isn't the patterns themselves — they're straightforward regex on familiar SDK idioms (`messages=[{"role": "system"...}]` , `stream=True` , `subprocess.run(tool_call.input.command)` ). The interesting part is the benchmark we ran while developing them, against the three other tools anyone would compare us to. ## The four tools, the honest map **Bandit** (PyCQA) is the Python-OSS standard security linter. Hand-written rules. Free, fast, no LLM. Python only. **Semgrep** is multi-language SAST with community rule packs. Hand-written rules. Free, fast, no LLM. Same product shape as getdebug. **vulnhuntr** (Protect AI, open source) is the stated category leader for AI-app static analysis. LLM-driven, Python-only, entry-point-detection based. **getdebug** is what we ship: pattern-based regex prefilters in JS/TS + Python now, plus an optional local-LLM SAST pass via Ollama (free, on-device) and a hosted LLM SAST pass via Claude (paid). The Python additions in 0.4.0 are the regex layer specifically. ## Test 1 — paired vulnerable/safe fixtures (10 files) We wrote 5 vulnerable + 5 safe Python AI-app fixtures, one pair per category. Same shape as our existing JS/TS corpus. All four tools ran on the same set. ``` Tool TP FP FN Precision Recall getdebug 5 1 0 83% 100% bandit 1 1 4 50% 20% semgrep 1 1 4 50% 20% vulnhuntr — — — (0 files selected; see below) ``` Bandit and Semgrep both fire on the `unsafe-tool-output` fixture via their generic `subprocess.run(shell=True)` rules — that's a true positive on the vulnerable variant. But they also fire on the **safe** variant of the same fixture, the one where the model output is mapped through an allowlist before reaching the shell: ``` # Safe pattern — Bandit + Semgrep both flag this as a FP ALLOWED = {"hosts": "cat /etc/hosts", "uptime": "uptime"} def handle(tool_call): cmd = ALLOWED.get(tool_call.input.tag) if not cmd: return "rejected" return subprocess.run(cmd, shell=True, capture_output=True).stdout ``` Neither tool knows that `cmd` came from a static dict, not from the model. They see `shell=True` and fire. getdebug's regex requires the `tool_call.input.X` / `block.input.X` reference to appear in the sink arg, which narrows the false-positive surface the generic SAST tools don't. Honesty update: the `scanPyShellToolArg` detector added in 0.5.5 now fires a `CRITICAL` on this same safe fixture — it sees `tool_call.input.tag` reaching a `shell=True` sink and can't tell the value was routed through the static allowlist first. Re-running the corpus on 0.5.5 scores getdebug at 5 TP / 1 FP / 0 FN (83% precision) on this set, not the clean sweep the original 0.4.0 numbers showed. Tightening this allowlist case is tracked. Both tools miss the other four behavioural categories entirely — pii-in-prompt, unsafe-role-merge, prompt-injection, unbounded-stream. They aren't designed for them; the rule packs don't contain patterns for `{"role": "system", "content": f"...{'$'}{name}..."}` . That's our addition. ## Test 2 — the real-world signal/noise check Synthetic fixtures lock the behaviour in. The real test for a security scanner is what it does on actual code. We ran all three (working) tools against [simonw/llm](https://github.com/simonw/llm) — Simon Willison's clean, well-maintained CLI for talking to LLMs, 49 Python files (commit 0d593ea; raw tool outputs in bench/realworld/). ``` Tool Total findings Signal bandit 1,261 1,228 are 'assert_used' (pytest); zero AI-app coverage semgrep 3 3 generic-SAST hits; zero AI-app coverage getdebug 7 6 AI-app findings (1 prompt-injection, 5 unbounded-stream) + 1 borderline role-merge ``` Bandit's 1,261 findings on this 49-file codebase is *almost entirely noise*: 1,228 of them are `assert_used` warnings on pytest assertions. This is a long-standing Bandit complaint — the default config flags every `assert` as a security smell because asserts disappear under `python -O` . For a production app where you opt-out of -O optimisation (which is almost everyone), this is pure noise. Semgrep's 3 findings are real but generic: an `exec()` usage flagged in `cli.py` (intentional for the plugin system), a missing-integrity attribute on a static HTML asset, and a non-literal import. None of these are AI-app specific. Six of getdebug's seven findings are AI-app categorized: one prompt-injection (the CLI's template feature concatenating two user-supplied strings) and five unbounded-stream hits in the OpenAI plugin (each `stream=True` with no `with` block or timeout in scope). Both arguable as TPs depending on threat model — this is a single-user CLI, so the prompt-injection between two CLI-user inputs is a stretch, and the library delegates stream-management to its consumer. The seventh, a role-merge on `{"role": role}` , is a borderline false positive — the role echoes the API's own streaming response, not request input — the kind of thing you'd triage away. Everything is flagged `HIGH` / `MEDIUM` , not `CRITICAL` , with context to triage. The CLI suppresses findings with a `.getdebug-ignore` file (inline `// getdebug:ignore` annotations are a hosted-scan feature). ## About vulnhuntr vulnhuntr is the stated category leader for LLM-driven AI-app static analysis, so we re-checked the current release. Two findings, and we want to be fair about both: **It runs.** vulnhuntr 1.2.2 installs and starts cleanly on Python 3.11 — the older`ModuleNotFoundError` is gone. Like any LLM-driven tool it needs a provider (an API key, or a local Ollama via`--llm ollama` ).**But it sees nothing here.** Its file-selection heuristic targets “network-exposed” entry points, and simonw/llm is a CLI, not a web app — so vulnhuntr selects**zero files** to analyse and surfaces nothing on this repo. Confirmed with its own`--dry-run` (0 files, $0.00, no key needed). So vulnhuntr isn't broken — it's scoped to web-app entry points, which makes it a no-op on a CLI library like this one. Different tool, different target: getdebug's AI-app prefilters run on any Python that touches an LLM, web app or not. That's the gap we're shipping into. ## What this means for you If you ship a Python app that calls an LLM — chat wrapper, agent framework, tool-calling backend, batch summariser — you should run *all three*: `bandit -r .` for general Python security hygiene (turn off`assert_used` via`.bandit` config first).`semgrep --config auto .` for cross-language SAST coverage.`npx @getdebug/cli@0.4.0 analyze .` for the AI-app behavioural patterns the other two don't target. They're complementary. None of them subsume the others. The first two catch general SAST and Python hygiene; getdebug catches the “serialised the whole user object into the LLM prompt” class of bugs that you can't hand-write a sustainable rule for in generic SAST without painful ergonomics. Reproduce every number on this page at [getdebug.dev/bench](/bench). Corpus, methodology, and harness are open — [CodeSecBench](https://github.com/getdebug-ai/codesecbench) on GitHub. And as always: if you see a result you can't reproduce, or a pattern getdebug should catch but doesn't, the corpus is open — PRs welcome, harness is reproducible, methodology is documented. That's the whole point of doing the benchmark in public. try it npx @getdebug/cli@0.4.0 analyze . Or via Homebrew: `brew install getdebug-ai/tap/getdebug` — Fafa Agbetsise / Founder, getdebug.dev