# Comparing Open-Source AI Code Security Harnesses > Source: > Published: 2026-07-20 00:00:00+00:00 There's a huge new wave of open-source AI security projects that point a large language model at a codebase to find vulnerabilities. We looked at some popular (or less popular, but from interesting companies) OSS tools and compared how they work, and when you'd reach for each. We find three complementary categories in the new world of harnesses: LLM-led exploitgen, LLM-skill-boosting, and SAST+LLM hybrids. **LLM-led exploitgen** For this category, the harness' goal is achieving to a crashing end-state as an oracle for vulnerabilities. I think of this as a new form of fuzzing. These have been the hardest harnesses to use in practice as model guardrails prevent LLMs from generating exploits, so you'll need to be part of the inner circle of trusted access/cyber verification. Additionally, they are usually very narrowly scoped, focusing on just a few different types of vulnerabilities with a reliable exploit condition. But there is a lot of low hanging fruit in that category! **LLM-skill-boosting vulnerability research** This approach adds skills *inside* LLMs to get them to reason about code the way a human vulnerability researcher would to identify vulnerabilities. __security-audit-skill__ (Cloudflare, 2K stars, MIT license): 6-phase multi-agent audit with adversarial validation __skills__ (Trail of Bits, 6K stars, CC-BY-SA license): ~40 plugins designed to distill the expert knowledge of a security consultant __vulnhunter__** **(CapitalOne, 500 stars, Apache license): 3 (find, fix, verify) composable Claude Code skills that form an automated remediation loop __mantis__ (Google, 400 stars, Apache license): ~15 security-focused skills. "Intended to be a starting point rather than a rigid set of instructions." **SAST+LLM hybrids** Using deterministic program analysis tools (e.g.: Semgrep, CodeQL) as inputs to LLMs to guide the search space. __raptor__ (Community, 3K stars, MIT license) __deepsec__ (Vercel Labs, 5K stars, Apache license) __ai-deep-sast____ __(Cisco, 50 stars, Apache license) __VVAH__ (Visa, 600 stars, Apache license) A broad, deeply-agentic 11-stage SAST pipeline (42 languages) that threat-models, verifies adversarially, then proposes and validates fixes (without executing code) ### Ideal use cases There's a lot of overlap between these harnesses, and it's not clear that all of them will continue to be maintained. Still, I did my best to come up with some categorization of when each of these might make the most sense. If you care about running fully local and offline: If you're a vulnerability hunter: **raptor** — one platform spanning scanning, dataflow, fuzzing, exploit feasibility, and SCA, if you have the operational appetite. **google/mantis – **if you are going to invest a lot of time and want a starting point to iterate and develop your own skills. If you're a C/C++ maintainer who wants high-confidence exploitable findings with verified patches: If you're running an appsec program and trying to optimize for time-to-reviewed-fix: **VVAH**: broad-language SAST with some real triage rigor (threat modeling, adversarial verify, exploit-chain synthesis, propose-and-validate-the-fix). Accepts a frontier-model dependency. No runnable PoC. If you're a Next.js/Vercel shop: If you are already using Claude Code / Codex / another coding harness for vulnerability hunting, why not install all the skills? **cloudflare/security-audit-skill** — you already live in Claude Code and want a rigorous, adversarially-validated audit methodology with near-zero setup and no new infra. **trailofbits/skills** — you want a *toolbox* of specialist reviewers (native code, crypto constant-time, smart contracts) to compose into your own workflow, inside Claude Code or Codex. **capitalone/vulnhunter** – explicitly optimized for Claude Code specifically ### What's similar and different across these harnesses? **Separation of vulnerability discovery from validation**: almost every LLM+SAST harness does this, with many explicitly using separate models to do so. But Exploitgen harnesses tend to use the same model to both discover and generate the exploit. **Blurring line between traditional static and dynamic analysis: **ai-deep-sast, VVAH, and deepsec reason completely statically. defending-code-harness and raptor can reason and also compile and execute binaries to validate. The latter is a new hybrid analysis that is much closer to how human vulnerability research looks. **Adversarial validation:** where a second independent agent tries to falsify each finding (Cloudflare's Phase 3, VVAH S6, ToB's fp-check, and defending-code-harness fresh-container grader). This seems widely adopted as a good way to increase accuracy, especially when having a different model attempt to disprove findings. **Language support is less of a differentiator: **traditional SAST tools took a lot of time to add language support but LLMs generally support almost all languages out of the box. Although exploit-gen class of harnesses is more narrow in language support, the bottleneck is more the language-specific toolchain to build a binary than the language itself. **The definition of "findings" varies** across the harnesses. E.g.: **Tool** | **Primary method** | **"Finding" is…** | |---| ai-deep-sast | Semgrep + LLM triage | A triaged static match | VVAH | 11-stage agentic LLM pipeline | A verified candidate from LLM pipeline | raptor | Semgrep+CodeQL+AFL+++LLM | A verified static finding or LLM pipeline candidate (± PoC) | defending-code-harness | LLM agents craft crashing inputs | A reproducible crash (ASAN) | deepsec | regex prefilter + AI + revalidate | A re-validated static match | **Patch generation is a cool capability but still less common: **LLM-led exploit generation tools are (obviously) executing code, producing POCs, and (less obviously) patches. Most of the SAST+LLM hybrids are not: **Tool** | **Executes code?** | **PoC** | **Patch** | |---| ai-deep-sast | No | ✗ | ✗ (advice) | VVAH | No | ✗ | ✓ propose + LLM-validate | raptor | Yes | ✓ (fuzz/feasibility) | ✓ | defending-code-harness | Yes | ✓ | ✓ execution-verified | deepsec | No | ✗ | ✗ | ### A market leader emerges? Will there be a reference open source LLM harness for the new era? We think not today, and likely many companies will build their own "shop jigs" (__thanks to tptacek on HN for the anology)__ for vulnerability finding. An OSS leader may emerge later, but the field is moving too fast right now. That's also why some of these harnesses are being released but marked as "no external contributions accepted" or unmaintained; the field is simply moving too fast. We at Semgrep do have something we're planning to announce here at Black Hat in August. Stay tuned! *The above was human-written; see below for LLM-generated summaries of some of the repos involved!* ### Reference 1: LLM summaries of the LLM+SAST tools #### ai-deep-sast (Cisco) *LLM summary: *Hybrid SAST in Python. Semgrep does fast rule-based detection; an LLM triages each finding behind an "evidence gate" meant to surface only true positives. A **fast scan** can run a fully local, security-tuned 8B model (Foundation-Sec-8B) so no code leaves the machine; a **deep scan** hands function-level context to a frontier model. Markdown/JSON/JUnit output, CI severity gates. Its optional fully-local mode is a genuine privacy differentiator. Brand new (v1.0.0). #### VVAH — visa-vulnerability-agentic-harness (Visa) *LLM summary:* An agentic SAST pipeline (built on learnings from Anthropic's Project Glasswing) running an 11-stage flow: threat-model (STRIDE) → decompose into taint chunks → deep-dive with optional majority voting → adversarially verify each finding (a second reviewer tries to falsify it, assigns CVSS) → dedup → synthesize exploit chains → SARIF → propose a minimal fix → score that fix with an agentic panel (security-architect + pen-tester + cross-repo-analyzer). Vendor-neutral backends; default Sonnet 4.6 for detection, Opus 4.8 for remediation/validation. 42 languages + 6 specialist lenses. No code execution. Optimizes "Mean Time to Adapt" (discovery → validated fix). Apache 2.0; closed to external contributions. #### raptor (community, MIT) *LLM summary: *The maximalist — ~107K lines of Python orchestrating Semgrep, CodeQL, AFL++ fuzzing, Z3-based exploit feasibility, SCA, and multi-model consensus, fronted by Claude Code with loadable "expert personas." A six-stage exploitability-validation pipeline filters false positives before investing in exploit dev. Broadest capability surface; heaviest to operate. #### defending-code-harness (Semgrep) *LLM summary: *Agentic, execution-verified, focused on **C/C++ memory-safety bugs**. Agents craft malicious inputs; a "find" only counts once it crashes under AddressSanitizer and **reproduces 3/3 times in an isolated container**. An independent grader re-confirms in a fresh container, an LLM judge dedups semantically, and a patch agent produces a fix verified on a **T0–T3 ladder** (compiles → PoC stops → tests pass → survives re-attack). gVisor sandbox + egress allowlist. Narrow scope, but findings are real crashes, not opinions. #### deepsec (Vercel Labs) *LLM summary: *TypeScript monorepo (~37K LOC). A 198-matcher regex prefilter (gated by auto-detected tech stack) finds candidate sites for free; an AI investigation loop then reasons over context and auth boundaries; a revalidation pass cuts false positives ~50%. Read-only agent tools (no execution), sandboxed with bubblewrap/Seatbelt locally or Vercel microVMs for distributed runs. Multi-backend: default Codex GPT-5.5, or Claude Opus 4.8, or Pi (GLM 5.2). Strong web/app-framework coverage (Next.js, Django, etc.) plus IaC. --diff mode for PR review; reports per-batch token cost. No SARIF yet; findings as JSON/Markdown, no patches. ### Reference 2: LLM-summaries of the LLM-skill-boosting group #### security-audit-skill (Cloudflare) *LLM summary: *A Claude Code skill: ~1,200 lines of Markdown methodology + a zero-dependency Node validator, installed via npx skills add. Runs a **6-phase** audit — recon → parallel hunt (8–12+ agents by attack class) → adversarial validation (separate agents try to *disprove* each finding) → report → schema-validated findings.json → independent Phase-6 verification against source. Language-agnostic; domain playbooks for memory-safety/binary, AI/LLM, web-protocol/auth, and client-side. May optionally build and run code to confirm. Output is a human report + a SARIF-like custom JSON schema (payloads and repro steps, not standalone exploits). MIT. The intelligence is whatever model hosts it; the skill is the methodology. #### trailofbits/skills (Trail of Bits) *LLM summary: *A marketplace of **~40 plugins (~15 security-focused)** for Claude Code and Codex (CC-BY-SA). Standouts: **c-review** and **rust-review** use a deterministic Python planner to orchestrate parallel Claude workers (haiku/sonnet/opus) with shared prompt caching, then sequential dedup/false-positive judges, emitting **SARIF 2.1.0**; **static-analysis** wraps Semgrep + CodeQL; **constant-time-analysis** and **zeroize-audit** drop to assembly/LLVM-IR to catch crypto timing leaks and compiler-eliminated secret wiping; **fp-check** and **variant-analysis** are triage/expansion methodologies. Deep bench for **native code, cryptography, and smart contracts** (6 blockchains). Strength is breadth and domain depth; it's a toolbox, not one pipeline. ### Reference 3: LLM-generated "Pipelines vs. skills, at a glance" | **Standalone pipelines** | **Agent-native skills** | |---| Deploy as | Installed program / CI job | Prompt pack inside Claude Code / Codex | Brings its own model calls | Yes | No — uses the host agent's model | Runs unattended in CI | Yes | Partially (agent must drive) | Sandbox | Its own (gVisor, microVM, OS-level) | Inherits the host agent's | Cost model | Per-run infra + tokens | Just tokens on your existing agent | Examples | ai-deep-sast, VVAH, raptor, DCH, deepsec | Cloudflare security-audit, ToB skills | ### Reference 4: LLM-generated Capability matrix "Skill" rows inherit host-agent behavior where noted. **Tool** | **Kind** | **Vuln scope** | **Languages** | **Output** | **Isolation** | **Default model(s)** | **License** | |---| ai-deep-sast | Pipeline | OWASP/CWE Top 25, secrets, AI/ML | 30+ | MD/JSON/JUnit | n/a (static) | Foundation-Sec-8B (local) / frontier | Apache 2.0 | VVAH | Pipeline | 77+ CWEs, 6 lenses | 42 | MD + SARIF | tool sandbox (no Bash) | Sonnet 4.6 / Opus 4.8 | Apache 2.0¹ | raptor | Pipeline | 22 classes: mem+web+auth+logic | many + binaries | JSON+SARIF+SBOM | Landlock+seccomp+ns | multi-model (Claude/GPT/…) | MIT | defending-code-harness | Pipeline | C/C++ memory safety | C/C++ (ext.) | SARIF+patches | gVisor + egress allowlist | Claude (Opus/Sonnet) | Apache 2.0 | deepsec | Pipeline | web/app + IaC + agentic-AI | 10+ | JSON/MD | bubblewrap/Seatbelt/microVM | Codex GPT-5.5 (or Claude/Pi) | see repo² | security-audit-skill | Skill | broad + native/AI/web/client playbooks | agnostic | MD + JSON schema | host agent's | host model | MIT | trailofbits/skills | Skills | native, crypto, smart-contract, +more | many + 6 chains | SARIF/MD/JSON | host agent's | haiku/sonnet/opus | CC-BY-SA 4.0 | ¹ closed to external contributions. ² confirm license in-repo before relying on it. ³ certain skills (e.g. constant-time, zeroize) compile/inspect assembly; most are read-only.