# Why You Need a Vulnerability Harness, Not a Security Agent > Source: > Published: 2026-07-10 08:02:28+00:00 [Security](https://sourcefeed.dev/c/security)Article # Why You Need a Vulnerability Harness, Not a Security Agent Stop pointing raw LLMs at your codebase. Build a stateful, model-agnostic pipeline to find and validate real security bugs. [Ji-ho Choi](https://sourcefeed.dev/u/jiho_choi) Pointing an LLM at a repository and asking it to find security bugs is a great way to burn through your API budget while generating a mountain of useless noise. If you have tried running a simple security-audit prompt or a basic coding agent on a production codebase, you have likely hit the same wall. Within an hour, the agent fills its context window, begins cannibalizing its own memory, forgets the vulnerabilities it was tracking, and eventually crashes due to a rate limit, forcing you to start from scratch. Even if the agent completes its run, it only finds a fraction of the actual bugs. A single model looking at a codebase acts like a single human reviewer: it operates under a specific set of cognitive biases. If you run the same prompt ten times, you might get ten different sets of half-baked findings that you have to manually deduplicate and verify. To move from localized security scripts to a continuous, fleet-wide scanning pipeline, you have to decouple the security logic from the underlying model. You do this by building a vulnerability harness. The harness is the durable, stateful infrastructure that manages state controls, eliminates false positives, and coordinates triage. The LLMs are treated as interchangeable, stateless compute engines. ## The Limits of Single-Session Agents The core issue with generic coding agents in security workflows is that they are fundamentally unsuited for the scale of vulnerability discovery. Security analysis requires running hundreds of separate investigations that must survive across runs, avoid sharing a single context window, and remain referenceable for downstream triage. When you run a multi-agent session inside a single context window, three things break: **Context Exhaustion:** As the model explores a repository, it accumulates directory structures, file contents, and call graphs. Eventually, the context window fills up. The model then compacts its context, discarding the subtle clues it needed to trace a taint path across multiple modules.**Brittle Persistence:** If your pipeline relies on a single long-running session, a single network hiccup or API rate-limit error destroys the entire state. Losing hours of analysis because of a transient API error is an expensive lesson in system design.**Cross-Repository Blindness:** Vulnerabilities rarely sit neatly inside a single file. They often lurk in the interfaces between components, where a library makes assumptions that the consuming application violates. A single-session agent is blind to these cross-repo dependencies. To solve this, you must externalize the state entirely. The harness should treat the LLM as a stateless function. Every step of the analysis must be written to a persistent database, allowing the pipeline to be paused, resumed, and deduplicated. ## Anatomy of a Multi-Stage Harness A production-grade vulnerability harness replaces a single conversational loop with a structured pipeline. Cloudflare recently detailed their transition from a 450-line security-audit script to a multi-stage harness, and Anthropic released an open-source reference implementation called the [Defending Code Reference Harness](https://github.com/anthropics/defending-code-reference-harness) that follows a similar architecture. A robust harness divides the work into distinct, isolated stages: ``` php flowchart TD A[Recon Stage] -->|Architecture Map| B[Hunt Stage] B -->|Raw Findings| C[Validate Stage] C -->|Candidate Vulnerabilities| D[Mechanical Validation] D -->|Schema-Valid Findings| E[Independent Verification] E -->|Verified Bugs| F[Ingest & Patch] ``` ### 1. Recon Before looking for bugs, the harness maps the target. Specialized recon agents analyze the directory structure, identify entry points, map APIs, and write an architecture map. This map acts as a guide for the rest of the pipeline, ensuring the hunting agents do not waste compute on test suites or build scripts. ### 2. Hunt Instead of a single generalist agent, the harness deploys multiple "Hunter" agents. Each hunter is configured to search for a specific class of vulnerability, such as memory corruption, injection, or broken access control. They use the architecture map to target high-risk areas like input parsers or authentication boundaries. ### 3. Validate Every raw finding from a hunter is immediately handed to an adversarial validator agent. The validator's sole job is to disprove the finding. It attempts to find sanitization paths, configuration guards, or compiler flags that would prevent the vulnerability from being exploited. If the validator can prove the bug is unreachable, the finding is discarded. ### 4. Mechanical Validation Before wasting more LLM tokens, the harness runs fast, deterministic checks on the remaining findings. It parses the target code to verify that the reported file paths, function names, and line numbers actually exist. This step filters out hallucinations before they reach human reviewers. ### 5. Independent Verification The surviving findings are handed to a fresh, isolated agent that has no access to the previous discussion history. This agent is given only the raw source code and the vulnerability report, and it must independently verify that the bug exists. If this independent check fails, the finding is flagged for manual review or discarded. ## The Developer Angle: Building a Minimal Harness You do not need a complex multi-agent framework to start. A minimal, highly effective harness can be built with a relational database, a task queue, and a few structured prompts. First, define a strict schema for your findings. This ensures that your agents emit data that can be parsed, deduplicated, and validated programmatically. Here is an example of a structured schema for a vulnerability finding: ``` { "$schema": "http://json-schema.org/draft-07/schema#", "title": "VulnerabilityFinding", "type": "object", "properties": { "id": { "type": "string", "format": "uuid" }, "vulnerability_type": { "type": "string" }, "severity": { "enum": ["CRITICAL", "HIGH", "MEDIUM", "LOW"] }, "file_path": { "type": "string" }, "function_name": { "type": "string" }, "line_number": { "type": "integer" }, "description": { "type": "string" }, "proof_of_concept": { "type": "string" } }, "required": ["id", "vulnerability_type", "severity", "file_path", "function_name", "line_number", "description"] } ``` To implement this, write a runner script that manages the state of each repository scan in a database like PostgreSQL. The runner should execute the stages sequentially, saving the output of each stage before moving to the next. ``` python # A conceptual runner managing state persistence import uuid import json def run_harness_pipeline(repo_id, db_connection): # 1. Recon recon_data = db_connection.get_recon(repo_id) if not recon_data: recon_data = run_recon_agent(repo_id) db_connection.save_recon(repo_id, recon_data) # 2. Hunt findings = db_connection.get_findings(repo_id) if not findings: raw_findings = run_hunt_agents(repo_id, recon_data) findings = save_and_deduplicate(db_connection, repo_id, raw_findings) # 3. Validate for finding in findings: if finding["status"] == "PENDING_VALIDATION": is_valid = run_adversarial_validator(repo_id, finding) status = "VALIDATED" if is_valid else "DISPROVED" db_connection.update_finding_status(finding["id"], status) ``` If your pipeline goes beyond static analysis and attempts to run the code to verify crashes, sandboxing is non-negotiable. Running LLM-generated exploits or compiling untrusted code on your host machine is a massive security risk. You should run your execution environment inside a secure sandbox like [gVisor](https://gvisor.dev) or a hardened [Docker](https://www.docker.com) container with strict network egress controls. For C/C++ codebases, compile the target with [AddressSanitizer](https://github.com/google/sanitizers/wiki/AddressSanitizer) (ASAN) and have your validation agent run the compiled binary with the generated proof-of-concept input to verify the crash programmatically. ## Why Model Agnosticism is Your Best Defense A common mistake is building a harness tightly coupled to a single model provider's SDK. The AI ecosystem shifts too quickly for this approach. The model that leads the benchmarks today might be superseded next month, or its API might suffer from latency spikes and rate limits that stall your pipeline. More importantly, relying on a single model limits your defensive coverage. When Cloudflare pointed different frontier models at the same target codebase, they found that each model turned up a different share of the bugs. The overlap was surprisingly small. By varying the models across your pipeline, you introduce cognitive diversity. For example, you might use a highly creative model with a large context window for the initial Recon and Hunt stages, and then use a highly precise, reasoning-focused model for the Validate and Independent Verification stages. This cross-checking ensures that systematic biases in one model's logic are caught by another. This architectural shift is supported by recent research. Systems like AgentFlow, which uses a typed graph DSL to coordinate multi-agent vulnerability discovery, have demonstrated that the coordination protocol and feedback loops matter far more than the raw capabilities of any single model. By using feedback-driven loops that read runtime signals from the target program, these harnesses have successfully discovered critical zero-day vulnerabilities in complex software like Google Chrome. ## The Durable Layer Models are transient. They are upgraded, deprecated, and replaced at a rapid pace. If you build your security tooling around a specific model or a set of fragile system prompts, your pipeline will break the moment the underlying model shifts. The harness is the piece that lasts. By building a stateful, database-backed orchestration layer that treats LLMs as interchangeable compute, you create a system that can adapt to new models as they arrive. You also build a pipeline that can scale from a single repository to an entire enterprise fleet, turning raw, noisy model outputs into a clean, actionable queue of verified security fixes. ## Sources & further reading - [Build your own vulnerability harness](https://blog.cloudflare.com/build-your-own-vulnerability-harness/)— blog.cloudflare.com - [Build your own vulnerability harness — CyberSecurity News | HackerFeeds](https://hackerfeeds.com/news/build-your-own-vulnerability-harness-zy6vz2)— hackerfeeds.com - [GitHub - anthropics/defending-code-reference-harness: Skills for threat modeling, scanning, triage, patching, plus an autonomous scanning harness you can /customize · GitHub](https://github.com/anthropics/defending-code-reference-harness)— github.com [Ji-ho Choi](https://sourcefeed.dev/u/jiho_choi)· Security & Cloud Editor Ji-ho covers the increasingly tangled overlap between cloud architecture and security, drawing on a background as a penetration tester to keep his reporting grounded in real-world attack paths. He never lets a vendor claim go unquestioned and insists that every buzzword come with a proof of concept. ## Discussion 0 No comments yet Be the first to weigh in.