# Ask HN: Which CVEs should I add to my Python security benchmark for AI agents?

> Source: <https://github.com/GiovanniGatti/cve-bench>
> Published: 2026-07-23 14:10:28+00:00

A benchmark for evaluating LLM agents on fixing real-world security vulnerabilities. Agents run inside sandboxed Docker containers and are scored against the maintainer's security test suite.

- Python 3.12+
- Docker
`OPENAI_API_KEY`

,`ANTHROPIC_API_KEY`

, and/or`POOLSIDE_API_KEY`

in your environment (or a`.env`

file)

Install dependencies:

```
pip install poetry
poetry install
```

Each task lives under `tasks/{CVE-ID}/`

and contains:

```
tasks/CVE-2026-33175/
├── meta.json           # GHSA ID, CWE, CVSS, repo URL, vulnerable and fixed SHAs
├── setup.sh            # Clones repo, checks out the vulnerable SHA, installs dependencies
├── run_tests.sh        # Injects test_security.py into the repo and runs pytest
├── test_security.py    # Security tests (xfail on vulnerable code, pass on the fix)
├── advisory.md         # Full GHSA advisory (richest prompt)
├── diagnose.md         # Behavioural description only — no file or function names
├── locate.md           # File and function only — no description of the flaw
└── Dockerfile          # Optional; only present when the task needs extra system deps
```

`meta.json`

example:

```
{
  "ghsa_id": "GHSA-xxxx-xxxx-xxxx",
  "cwe": ["CWE-287"],
  "cvss": 9.1,
  "repo": {
    "url": "https://github.com/org/project",
    "vulnerable_sha": "abc123^",
    "fixed_sha": "abc123"
  }
}
```

`setup.sh`

is idempotent and safe to re-run. `test_security.py`

is kept hidden from the agent during the run and injected only after the agent finishes.

```
python build.py
```

This builds:

- A shared base image (
`cve-bench/base`

) — Python 3.12, git, poetry, and the harness. - One task image per task (
`cve-bench/{task-id}`

) — extends the base, copies the task directory, and runs`setup.sh`

.

**Options:**

```
# Build specific tasks only
python build.py --task CVE-2026-33175 CVE-2026-42561

# Skip rebuilding the base image
python build.py --skip-base
```

Task images are built in parallel (up to 5 workers). If a task directory contains a `Dockerfile`

, it is used instead of the generic `docker/task.Dockerfile`

.

Before running the benchmark, verify that each task's security tests correctly distinguish vulnerable from fixed code:

```
python validate.py
```

For each task, this runs three phases inside the task container:

| Phase | What it checks |
|---|---|
vulnerable |
Security tests must fail (or xfail) on the vulnerable SHA |
fixed |
Security tests must pass on the fixed SHA |
regression |
Non-security tests must pass on the fixed SHA |

Results are displayed as a live table. Exit code is 1 if any task fails any phase.

```
# Validate specific tasks only
python validate.py --task CVE-2026-33175 GHSA-r758-8hxw-4845

# Skip rebuilding images before validation
python validate.py --skip-build
python benchmark.py --model openai:gpt-5.5 poolside:laguna-m.1 --prompt-type advisory
```

**Options:**

| Flag | Description | Default |
|---|---|---|
`--model` |
One or more `provider:model-id` strings |
all configured models |
`--prompt-type` |
`advisory` , `diagnose` , `locate` , or any combination |
all three |
`--task` |
One or more task IDs | all tasks |
`--clean` |
Delete existing results for the selected scope before starting | off |

**Supported providers:**

| Provider | Format | API key env var |
|---|---|---|
| OpenAI | `openai:gpt-5.5` |
`OPENAI_API_KEY` |
| Anthropic | `anthropic:claude-haiku-4-5-20251001` |
`ANTHROPIC_API_KEY` |
| Poolside | `poolside:laguna-m.1` |
`POOLSIDE_API_KEY` |

Each run produces a JSON result file in `results/`

:

```
results/{task-id}__{provider}:{model}__{prompt-type}.json
```

Existing result files are skipped automatically. Runs execute concurrently across tasks (up to 20 workers), with per-provider rate limiting (one active request per provider at a time) to avoid 429s.

Each result file is a JSON object with the following structure:

```
{
  "cve_id": "CVE-2026-33175",
  "model_id": "openai:gpt-5.5",
  "prompt_type": "advisory",
  "timestamp": "2026-05-01T12:00:00",
  "model_duration_s": 142.3,
  "test_duration_s": 8.1,
  "turns": [
    {
      "tool_calls_and_results": [...],
      "input_tokens": 12400,
      "output_tokens": 310
    }
  ],
  "tests": [
    {
      "kind": "security",
      "name": "test_email_verified",
      "outcome": "passed"
    }
  ]
}
```

`tests[].kind`

is either `"security"`

(from `test_security.py`

) or `"regression"`

(from the project's own test suite). A run is considered solved only if all security tests pass and no regression tests fail.

```
python generate_charts.py
```

Reads all result files from `results/`

and writes SVG charts to `docs/images/charts/`

. Requires Chrome/Chromium for Bokeh's headless export (via `chromedriver-binary`

).

The harness runs inside each Docker container as `python -m harness.run`

. It is responsible for loading the prompt, running the agentic loop, and writing the result file.

```
src/harness/
├── run.py                  # Entry point; parses args, wires components, calls BenchmarkRunner
├── client/
│   ├── factory.py          # Parses provider:model-id, returns the correct LLMClient
│   ├── _client.py          # Abstract LLMClient, ToolCall and LLMTurn dataclasses
│   ├── anthropic.py        # Anthropic SDK integration
│   └── oai.py              # OpenAI SDK integration (also used for Poolside)
├── agent/
│   ├── core.py             # Agentic loop: calls client, dispatches tool calls, threads messages
│   └── runner.py           # Wraps Agent, tracks timing and turn list
├── bench/
│   ├── runner.py           # Orchestrates setup → agent → security tests → regression tests
│   ├── result.py           # BenchmarkResult and TestResult dataclasses, JSON serialisation
│   └── repository.py       # Writes result files to disk
└── task/
    ├── tools.py             # Tool implementations: ListFiles, ReadFile, SearchInFiles,
    │                        #   EditFile, CreateFile, DeleteFile, RunPytest
    └── prompt_loader.py     # Reads advisory.md / diagnose.md / locate.md
```

**Tools available to the agent:**

| Tool | Description |
|---|---|
`list_files` |
List files and directories in the repository |
`read_file` |
Read file contents, optionally a line range |
`search_in_files` |
Regex search across the codebase with optional file glob |
`edit_file` |
Replace a range of lines in an existing file |
`create_file` |
Create a new file |
`delete_file` |
Delete a file |
`run_pytest` |
Run the project's test suite; returns a JSON report |

All tools validate paths against the repository root to prevent directory traversal. The agent does not have access to `test_security.py`

or to the git history.

The agent loop runs for at most 20 turns. If the turn ceiling is reached, the run is recorded as-is and the security tests are still executed against whatever state the agent left the repository in.

- Create
`tasks/{CVE-ID}/`

and add`meta.json`

,`setup.sh`

,`run_tests.sh`

,`test_security.py`

,`advisory.md`

,`diagnose.md`

,`locate.md`

. - Make
`setup.sh`

and`run_tests.sh`

executable (`chmod +x`

). - Validate:
`python validate.py --task {CVE-ID}`

. - Build:
`python build.py --task {CVE-ID}`

.

This work was conducted as independent research. At the time of conducting the research and preparing this repository, I had no institutional affiliation.

```
@misc{gattipinheiro2026cvebench,
  author       = {Gatti Pinheiro, Giovanni},
  title        = {{CVE-Bench}: Benchmarking {LLM} Agents on Real-World Security Vulnerability Fixes},
  year         = {2026},
  howpublished = {\url{https://giovannigatti.github.io/cve-bench}},
  note         = {Code available at \url{https://github.com/GiovanniGatti/cve-bench}}
}
```

MIT — see [LICENSE](/GiovanniGatti/cve-bench/blob/main/LICENSE).