If you're shipping anything with an LLM in the request path (a support bot, an agent that calls tools, a RAG pipeline reading untrusted documents) you eventually have to answer "did we actually test this for prompt injection, or did we just eyeball it." There are a handful of open source tools that can help, and they are not interchangeable. Picking the wrong one wastes a day; picking none wastes a lot more than that. I went through the five tools that come up most often, ran them where I could, and read the source/docs closely where I couldn't. Here's what each one actually does, where they overlap, and where they don't.
Update, Aug 2026: Microsoft archived PyRIT on GitHub on March 27, 2026. It's read-only now, no commits, no releases, no issue triage. I've kept the section below for context since a lot of existing guides and tutorials still point people to it, but if you're picking a tool today, skip to the "which one, for what" section, or read the dedicated post on what to use instead.
Most confusion here comes from mixing up two different things you can test:
promptfoo, Giskard, and sentinel-scan-cli are primarily app-layer tools. garak is model-layer. PyRIT can do either depending on how you configure it. Knowing which layer you actually need to test decides most of this for you before you even look at features.
App-layer, and the most broadly adopted of the bunch. Zero-install via npx promptfoo
, 50+ built-in red-team plugins, and it ships report presets mapped to OWASP LLM Top 10, NIST, and MITRE ATLAS out of the box. It's used internally at places like OpenAI and Anthropic (per their own repo), which tells you it holds up at scale, not just for toy demos.
Use it when: you want the most complete free coverage of your actual application, you're willing to spend real setup time wiring it into your prompts/agent config, and you want output that maps cleanly to a compliance framework someone above you is going to ask about. Tradeoff: the plugin surface is large enough that getting a useful first run takes more than five minutes. It rewards investment; it doesn't do much for you in a five-minute check.
Model-layer. Pip installable, 50+ probes, 8.1k GitHub stars, actively maintained by NVIDIA. It's testing the model's own susceptibility to jailbreaks, encoding tricks, data leakage, etc., not how your specific prompts and app logic hold up.
Use it when: you're evaluating which base model to build on, or you maintain a model and need to know its raw failure modes before anyone wraps an app around it. It is not natively organized around OWASP LLM Top 10, so if a compliance mapping is the deliverable you need, you'll be doing that translation yourself. Don't use it as your only test if what you actually ship is an application with system prompts, tool access, and retrieval, because none of that is in scope for a model-layer scan.
App-layer, and it does have OWASP-mapped detectors, similar spirit to promptfoo. The catch is that the continuous-scan Hub, the part that would let you run this repeatedly against a live app over time, is a paid product. The open source scanner itself is real and usable, but plan for the free tier to be a point-in-time check, not a monitoring setup.
Use it when: you want an OWASP-mapped scan and you're open to (or already have) their commercial tooling for anything ongoing. Status as of March 27, 2026: archived by Microsoft. The repo is read-only, no commits, no releases, no issue triage. Whatever version you have installed is the last one you'll get. Everything below describes what it did while it was maintained.
Multi-turn attack orchestration, built for red teamers who need to script conversational attack sequences (escalating a jailbreak over several turns, chaining techniques). It was more of a framework than a turnkey scanner, and it had noticeably lower general-developer adoption than the other four, mostly because the audience was security researchers running structured red-team engagements, not app developers doing a pre-ship check.
It doesn't have a direct one-for-one replacement for its multi-turn orchestration model. If that's specifically what you need, promptfoo's red-team plugins cover a lot of the same ground for app-layer testing, or you're looking at building a small harness yourself on top of a maintained model API.
Full disclosure: I work on this one, so weigh that accordingly, and I've tried to be honest about where it does and doesn't fit rather than pretend it's the answer for everyone.
It's a small, app-layer, dependency-free CLI (Python and Node, byte-identical output between the two) covering 15 attack patterns, each individually tagged with its OWASP LLM Top 10 category in both the console and JSON output. The whole pitch is speed to first result: --demo
runs with zero config and no API keys, and you get an OWASP-tagged pass/fail readout in under a minute.
Where it's genuinely useful: a fast, zero-setup first pass before you reach for something heavier, or a CI gate check when you want a quick OWASP-mapped signal without standing up a bigger red-team framework. Where it isn't: it covers 15 attacks, not 50+. If you need broad coverage or you're past the "quick check" stage and building an actual ongoing red-team practice, promptfoo or Giskard cover more ground and you should use those instead.
None of these are mutually exclusive. A reasonable setup for a team actually shipping an LLM app: sentinel-scan-cli or a promptfoo quick-start as a CI gate, promptfoo's full plugin set for a pre-launch pass, and garak if you're also evaluating model choice. Pick based on which layer you're actually testing and how much setup time you have, not by which one has the flashiest README.