Show HN: AI codebase analyser and auto-fixer

Getdebug CLI 0.4.0 adds Python AI-app regex prefilters for five categories (prompt-injection, unsafe-role-merge, pii-in-prompt, unbounded-stream, unsafe-tool-output), running in milliseconds with no LLM call. In benchmark tests against Bandit, Semgrep, and vulnhuntr on 10 paired vulnerable/safe Python AI-app fixtures, getdebug achieved 83% precision and 100% recall, while Bandit and Semgrep each scored 50% precision and 20% recall; vulnhuntr selected zero files. On the real-world repo simonw/llm, getdebug produced 1,261 findings with high signal, compared to Bandit's 1,261 and Semgrep's 1,261.

getdebug CLI 0.4.0 ships Python AI-app regex prefilters alongside the JS/TS ones that landed in 0.3.0. Five categories: prompt-injection, unsafe-role-merge, pii-in-prompt, unbounded-stream, unsafe-tool-output. All deterministic. No LLM call. The default getdebug analyze . on a Python repo runs in milliseconds, costs zero, and now covers Python AI-app anti-patterns alongside the secrets + dependency-CVE passes it always ran. The interesting part isn't the patterns themselves — they're straightforward regex on familiar SDK idioms messages= {"role": "system"...} , stream=True , subprocess.run tool call.input.command . The interesting part is the benchmark we ran while developing them, against the three other tools anyone would compare us to. The four tools, the honest map Bandit PyCQA is the Python-OSS standard security linter. Hand-written rules. Free, fast, no LLM. Python only. Semgrep is multi-language SAST with community rule packs. Hand-written rules. Free, fast, no LLM. Same product shape as getdebug. vulnhuntr Protect AI, open source is the stated category leader for AI-app static analysis. LLM-driven, Python-only, entry-point-detection based. getdebug is what we ship: pattern-based regex prefilters in JS/TS + Python now, plus an optional local-LLM SAST pass via Ollama free, on-device and a hosted LLM SAST pass via Claude paid . The Python additions in 0.4.0 are the regex layer specifically. Test 1 — paired vulnerable/safe fixtures 10 files We wrote 5 vulnerable + 5 safe Python AI-app fixtures, one pair per category. Same shape as our existing JS/TS corpus. All four tools ran on the same set. Tool TP FP FN Precision Recall getdebug 5 1 0 83% 100% bandit 1 1 4 50% 20% semgrep 1 1 4 50% 20% vulnhuntr — — — 0 files selected; see below Bandit and Semgrep both fire on the unsafe-tool-output fixture via their generic subprocess.run shell=True rules — that's a true positive on the vulnerable variant. But they also fire on the safe variant of the same fixture, the one where the model output is mapped through an allowlist before reaching the shell: Safe pattern — Bandit + Semgrep both flag this as a FP ALLOWED = {"hosts": "cat /etc/hosts", "uptime": "uptime"} def handle tool call : cmd = ALLOWED.get tool call.input.tag if not cmd: return "rejected" return subprocess.run cmd, shell=True, capture output=True .stdout Neither tool knows that cmd came from a static dict, not from the model. They see shell=True and fire. getdebug's regex requires the tool call.input.X / block.input.X reference to appear in the sink arg, which narrows the false-positive surface the generic SAST tools don't. Honesty update: the scanPyShellToolArg detector added in 0.5.5 now fires a CRITICAL on this same safe fixture — it sees tool call.input.tag reaching a shell=True sink and can't tell the value was routed through the static allowlist first. Re-running the corpus on 0.5.5 scores getdebug at 5 TP / 1 FP / 0 FN 83% precision on this set, not the clean sweep the original 0.4.0 numbers showed. Tightening this allowlist case is tracked. Both tools miss the other four behavioural categories entirely — pii-in-prompt, unsafe-role-merge, prompt-injection, unbounded-stream. They aren't designed for them; the rule packs don't contain patterns for {"role": "system", "content": f"...{'$'}{name}..."} . That's our addition. Test 2 — the real-world signal/noise check Synthetic fixtures lock the behaviour in. The real test for a security scanner is what it does on actual code. We ran all three working tools against simonw/llm https://github.com/simonw/llm — Simon Willison's clean, well-maintained CLI for talking to LLMs, 49 Python files commit 0d593ea; raw tool outputs in bench/realworld/ . Tool Total findings Signal bandit 1,261 1,228 are 'assert used' pytest ; zero AI-app coverage semgrep 3 3 generic-SAST hits; zero AI-app coverage getdebug 7 6 AI-app findings 1 prompt-injection, 5 unbounded-stream + 1 borderline role-merge Bandit's 1,261 findings on this 49-file codebase is almost entirely noise : 1,228 of them are assert used warnings on pytest assertions. This is a long-standing Bandit complaint — the default config flags every assert as a security smell because asserts disappear under python -O . For a production app where you opt-out of -O optimisation which is almost everyone , this is pure noise. Semgrep's 3 findings are real but generic: an exec usage flagged in cli.py intentional for the plugin system , a missing-integrity attribute on a static HTML asset, and a non-literal import. None of these are AI-app specific. Six of getdebug's seven findings are AI-app categorized: one prompt-injection the CLI's template feature concatenating two user-supplied strings and five unbounded-stream hits in the OpenAI plugin each stream=True with no with block or timeout in scope . Both arguable as TPs depending on threat model — this is a single-user CLI, so the prompt-injection between two CLI-user inputs is a stretch, and the library delegates stream-management to its consumer. The seventh, a role-merge on {"role": role} , is a borderline false positive — the role echoes the API's own streaming response, not request input — the kind of thing you'd triage away. Everything is flagged HIGH / MEDIUM , not CRITICAL , with context to triage. The CLI suppresses findings with a .getdebug-ignore file inline // getdebug:ignore annotations are a hosted-scan feature . About vulnhuntr vulnhuntr is the stated category leader for LLM-driven AI-app static analysis, so we re-checked the current release. Two findings, and we want to be fair about both: It runs. vulnhuntr 1.2.2 installs and starts cleanly on Python 3.11 — the older ModuleNotFoundError is gone. Like any LLM-driven tool it needs a provider an API key, or a local Ollama via --llm ollama . But it sees nothing here. Its file-selection heuristic targets “network-exposed” entry points, and simonw/llm is a CLI, not a web app — so vulnhuntr selects zero files to analyse and surfaces nothing on this repo. Confirmed with its own --dry-run 0 files, $0.00, no key needed . So vulnhuntr isn't broken — it's scoped to web-app entry points, which makes it a no-op on a CLI library like this one. Different tool, different target: getdebug's AI-app prefilters run on any Python that touches an LLM, web app or not. That's the gap we're shipping into. What this means for you If you ship a Python app that calls an LLM — chat wrapper, agent framework, tool-calling backend, batch summariser — you should run all three : bandit -r . for general Python security hygiene turn off assert used via .bandit config first . semgrep --config auto . for cross-language SAST coverage. npx @getdebug/cli@0.4.0 analyze . for the AI-app behavioural patterns the other two don't target. They're complementary. None of them subsume the others. The first two catch general SAST and Python hygiene; getdebug catches the “serialised the whole user object into the LLM prompt” class of bugs that you can't hand-write a sustainable rule for in generic SAST without painful ergonomics. Reproduce every number on this page at getdebug.dev/bench /bench . Corpus, methodology, and harness are open — CodeSecBench https://github.com/getdebug-ai/codesecbench on GitHub. And as always: if you see a result you can't reproduce, or a pattern getdebug should catch but doesn't, the corpus is open — PRs welcome, harness is reproducible, methodology is documented. That's the whole point of doing the benchmark in public. try it npx @getdebug/cli@0.4.0 analyze . Or via Homebrew: brew install getdebug-ai/tap/getdebug — Fafa Agbetsise / Founder, getdebug.dev