getdebug CLI 0.4.0 ships Python AI-app regex prefilters alongside the JS/TS ones that landed in 0.3.0. Five categories: prompt-injection, unsafe-role-merge, pii-in-prompt, unbounded-stream, unsafe-tool-output. All deterministic. No LLM call. The default getdebug analyze .
on a Python repo runs in milliseconds, costs zero, and now covers Python AI-app anti-patterns alongside the secrets + dependency-CVE passes it always ran.
The interesting part isn't the patterns themselves — they're straightforward regex on familiar SDK idioms (messages=[{"role": "system"...}]
, stream=True
, subprocess.run(tool_call.input.command)
). The interesting part is the benchmark we ran while developing them, against the three other tools anyone would compare us to.
The four tools, the honest map #
Bandit (PyCQA) is the Python-OSS standard security linter. Hand-written rules. Free, fast, no LLM. Python only.
Semgrep is multi-language SAST with community rule packs. Hand-written rules. Free, fast, no LLM. Same product shape as getdebug.
vulnhuntr (Protect AI, open source) is the stated category leader for AI-app static analysis. LLM-driven, Python-only, entry-point-detection based.
getdebug is what we ship: pattern-based regex prefilters in JS/TS + Python now, plus an optional local-LLM SAST pass via Ollama (free, on-device) and a hosted LLM SAST pass via Claude (paid). The Python additions in 0.4.0 are the regex layer specifically.
Test 1 — paired vulnerable/safe fixtures (10 files) #
We wrote 5 vulnerable + 5 safe Python AI-app fixtures, one pair per category. Same shape as our existing JS/TS corpus. All four tools ran on the same set.
Tool TP FP FN Precision Recall
getdebug 5 1 0 83% 100%
bandit 1 1 4 50% 20%
semgrep 1 1 4 50% 20%
vulnhuntr — — — (0 files selected; see below)
Bandit and Semgrep both fire on the unsafe-tool-output
fixture via their generic subprocess.run(shell=True)
rules — that's a true positive on the vulnerable variant. But they also fire on the safe variant of the same fixture, the one where the model output is mapped through an allowlist before reaching the shell:
ALLOWED = {"hosts": "cat /etc/hosts", "uptime": "uptime"}
def handle(tool_call):
cmd = ALLOWED.get(tool_call.input.tag)
if not cmd: return "rejected"
return subprocess.run(cmd, shell=True, capture_output=True).stdout
Neither tool knows that cmd
came from a static dict, not from the model. They see shell=True
and fire. getdebug's regex requires the tool_call.input.X
/ block.input.X
reference to appear in the sink arg, which narrows the false-positive surface the generic SAST tools don't. Honesty update: the scanPyShellToolArg
detector added in 0.5.5 now fires a CRITICAL
on this same safe fixture — it sees tool_call.input.tag
reaching a shell=True
sink and can't tell the value was routed through the static allowlist first. Re-running the corpus on 0.5.5 scores getdebug at 5 TP / 1 FP / 0 FN (83% precision) on this set, not the clean sweep the original 0.4.0 numbers showed. Tightening this allowlist case is tracked.
Both tools miss the other four behavioural categories entirely — pii-in-prompt, unsafe-role-merge, prompt-injection, unbounded-stream. They aren't designed for them; the rule packs don't contain patterns for {"role": "system", "content": f"...{'$'}{name}..."}
. That's our addition.
Test 2 — the real-world signal/noise check #
Synthetic fixtures lock the behaviour in. The real test for a security scanner is what it does on actual code. We ran all three (working) tools against simonw/llm — Simon Willison's clean, well-maintained CLI for talking to LLMs, 49 Python files (commit 0d593ea; raw tool outputs in bench/realworld/).
Tool Total findings Signal
bandit 1,261 1,228 are 'assert_used' (pytest);
zero AI-app coverage
semgrep 3 3 generic-SAST hits;
zero AI-app coverage
getdebug 7 6 AI-app findings (1 prompt-injection,
5 unbounded-stream) + 1 borderline role-merge
Bandit's 1,261 findings on this 49-file codebase is almost entirely noise: 1,228 of them are assert_used
warnings on pytest assertions. This is a long-standing Bandit complaint — the default config flags every assert
as a security smell because asserts disappear under python -O
. For a production app where you opt-out of -O optimisation (which is almost everyone), this is pure noise.
Semgrep's 3 findings are real but generic: an exec()
usage flagged in cli.py
(intentional for the plugin system), a missing-integrity attribute on a static HTML asset, and a non-literal import. None of these are AI-app specific.
Six of getdebug's seven findings are AI-app categorized: one prompt-injection (the CLI's template feature concatenating two user-supplied strings) and five unbounded-stream hits in the OpenAI plugin (each stream=True
with no with
block or timeout in scope). Both arguable as TPs depending on threat model — this is a single-user CLI, so the prompt-injection between two CLI-user inputs is a stretch, and the library delegates stream-management to its consumer. The seventh, a role-merge on {"role": role}
, is a borderline false positive — the role echoes the API's own streaming response, not request input — the kind of thing you'd triage away. Everything is flagged HIGH
/ MEDIUM
, not CRITICAL
, with context to triage. The CLI suppresses findings with a .getdebug-ignore
file (inline // getdebug:ignore
annotations are a hosted-scan feature).
About vulnhuntr #
vulnhuntr is the stated category leader for LLM-driven AI-app static analysis, so we re-checked the current release. Two findings, and we want to be fair about both:
It runs. vulnhuntr 1.2.2 installs and starts cleanly on Python 3.11 — the olderModuleNotFoundError
is gone. Like any LLM-driven tool it needs a provider (an API key, or a local Ollama via--llm ollama
).But it sees nothing here. Its file-selection heuristic targets “network-exposed” entry points, and simonw/llm is a CLI, not a web app — so vulnhuntr selectszero files to analyse and surfaces nothing on this repo. Confirmed with its own--dry-run
(0 files, $0.00, no key needed).
So vulnhuntr isn't broken — it's scoped to web-app entry points, which makes it a no-op on a CLI library like this one. Different tool, different target: getdebug's AI-app prefilters run on any Python that touches an LLM, web app or not. That's the gap we're shipping into.
What this means for you #
If you ship a Python app that calls an LLM — chat wrapper, agent framework, tool-calling backend, batch summariser — you should run all three:
bandit -r .
for general Python security hygiene (turn offassert_used
via.bandit
config first).semgrep --config auto .
for cross-language SAST coverage.npx @getdebug/cli@0.4.0 analyze .
for the AI-app behavioural patterns the other two don't target.
They're complementary. None of them subsume the others. The first two catch general SAST and Python hygiene; getdebug catches the “serialised the whole user object into the LLM prompt” class of bugs that you can't hand-write a sustainable rule for in generic SAST without painful ergonomics.
Reproduce every number on this page at getdebug.dev/bench. Corpus, methodology, and harness are open — CodeSecBench on GitHub.
And as always: if you see a result you can't reproduce, or a pattern getdebug should catch but doesn't, the corpus is open — PRs welcome, harness is reproducible, methodology is documented. That's the whole point of doing the benchmark in public.
try it
npx @getdebug/cli@0.4.0 analyze .
Or via Homebrew: brew install getdebug-ai/tap/getdebug
— Fafa Agbetsise / Founder, getdebug.dev