Ask HN: Which CVEs should I add to my Python security benchmark for AI agents?

A new open-source benchmark called CVE-Bench evaluates large language model agents on fixing real-world security vulnerabilities by running them inside sandboxed Docker containers and scoring them against maintainer security test suites. The benchmark supports Python 3.12+, Docker, and models from OpenAI, Anthropic, and Poolside, with each task containing a vulnerable SHA, fixed SHA, and multiple prompt types including advisory, diagnose, and locate. The project includes build, validation, and benchmarking scripts that produce JSON result files for each model and prompt combination.

A benchmark for evaluating LLM agents on fixing real-world security vulnerabilities. Agents run inside sandboxed Docker containers and are scored against the maintainer's security test suite. - Python 3.12+ - Docker OPENAI API KEY , ANTHROPIC API KEY , and/or POOLSIDE API KEY in your environment or a .env file Install dependencies: pip install poetry poetry install Each task lives under tasks/{CVE-ID}/ and contains: tasks/CVE-2026-33175/ ├── meta.json GHSA ID, CWE, CVSS, repo URL, vulnerable and fixed SHAs ├── setup.sh Clones repo, checks out the vulnerable SHA, installs dependencies ├── run tests.sh Injects test security.py into the repo and runs pytest ├── test security.py Security tests xfail on vulnerable code, pass on the fix ├── advisory.md Full GHSA advisory richest prompt ├── diagnose.md Behavioural description only — no file or function names ├── locate.md File and function only — no description of the flaw └── Dockerfile Optional; only present when the task needs extra system deps meta.json example: { "ghsa id": "GHSA-xxxx-xxxx-xxxx", "cwe": "CWE-287" , "cvss": 9.1, "repo": { "url": "https://github.com/org/project", "vulnerable sha": "abc123^", "fixed sha": "abc123" } } setup.sh is idempotent and safe to re-run. test security.py is kept hidden from the agent during the run and injected only after the agent finishes. python build.py This builds: - A shared base image cve-bench/base — Python 3.12, git, poetry, and the harness. - One task image per task cve-bench/{task-id} — extends the base, copies the task directory, and runs setup.sh . Options: Build specific tasks only python build.py --task CVE-2026-33175 CVE-2026-42561 Skip rebuilding the base image python build.py --skip-base Task images are built in parallel up to 5 workers . If a task directory contains a Dockerfile , it is used instead of the generic docker/task.Dockerfile . Before running the benchmark, verify that each task's security tests correctly distinguish vulnerable from fixed code: python validate.py For each task, this runs three phases inside the task container: | Phase | What it checks | |---|---| vulnerable | Security tests must fail or xfail on the vulnerable SHA | fixed | Security tests must pass on the fixed SHA | regression | Non-security tests must pass on the fixed SHA | Results are displayed as a live table. Exit code is 1 if any task fails any phase. Validate specific tasks only python validate.py --task CVE-2026-33175 GHSA-r758-8hxw-4845 Skip rebuilding images before validation python validate.py --skip-build python benchmark.py --model openai:gpt-5.5 poolside:laguna-m.1 --prompt-type advisory Options: | Flag | Description | Default | |---|---|---| --model | One or more provider:model-id strings | all configured models | --prompt-type | advisory , diagnose , locate , or any combination | all three | --task | One or more task IDs | all tasks | --clean | Delete existing results for the selected scope before starting | off | Supported providers: | Provider | Format | API key env var | |---|---|---| | OpenAI | openai:gpt-5.5 | OPENAI API KEY | | Anthropic | anthropic:claude-haiku-4-5-20251001 | ANTHROPIC API KEY | | Poolside | poolside:laguna-m.1 | POOLSIDE API KEY | Each run produces a JSON result file in results/ : results/{task-id} {provider}:{model} {prompt-type}.json Existing result files are skipped automatically. Runs execute concurrently across tasks up to 20 workers , with per-provider rate limiting one active request per provider at a time to avoid 429s. Each result file is a JSON object with the following structure: { "cve id": "CVE-2026-33175", "model id": "openai:gpt-5.5", "prompt type": "advisory", "timestamp": "2026-05-01T12:00:00", "model duration s": 142.3, "test duration s": 8.1, "turns": { "tool calls and results": ... , "input tokens": 12400, "output tokens": 310 } , "tests": { "kind": "security", "name": "test email verified", "outcome": "passed" } } tests .kind is either "security" from test security.py or "regression" from the project's own test suite . A run is considered solved only if all security tests pass and no regression tests fail. python generate charts.py Reads all result files from results/ and writes SVG charts to docs/images/charts/ . Requires Chrome/Chromium for Bokeh's headless export via chromedriver-binary . The harness runs inside each Docker container as python -m harness.run . It is responsible for loading the prompt, running the agentic loop, and writing the result file. src/harness/ ├── run.py Entry point; parses args, wires components, calls BenchmarkRunner ├── client/ │ ├── factory.py Parses provider:model-id, returns the correct LLMClient │ ├── client.py Abstract LLMClient, ToolCall and LLMTurn dataclasses │ ├── anthropic.py Anthropic SDK integration │ └── oai.py OpenAI SDK integration also used for Poolside ├── agent/ │ ├── core.py Agentic loop: calls client, dispatches tool calls, threads messages │ └── runner.py Wraps Agent, tracks timing and turn list ├── bench/ │ ├── runner.py Orchestrates setup → agent → security tests → regression tests │ ├── result.py BenchmarkResult and TestResult dataclasses, JSON serialisation │ └── repository.py Writes result files to disk └── task/ ├── tools.py Tool implementations: ListFiles, ReadFile, SearchInFiles, │ EditFile, CreateFile, DeleteFile, RunPytest └── prompt loader.py Reads advisory.md / diagnose.md / locate.md Tools available to the agent: | Tool | Description | |---|---| list files | List files and directories in the repository | read file | Read file contents, optionally a line range | search in files | Regex search across the codebase with optional file glob | edit file | Replace a range of lines in an existing file | create file | Create a new file | delete file | Delete a file | run pytest | Run the project's test suite; returns a JSON report | All tools validate paths against the repository root to prevent directory traversal. The agent does not have access to test security.py or to the git history. The agent loop runs for at most 20 turns. If the turn ceiling is reached, the run is recorded as-is and the security tests are still executed against whatever state the agent left the repository in. - Create tasks/{CVE-ID}/ and add meta.json , setup.sh , run tests.sh , test security.py , advisory.md , diagnose.md , locate.md . - Make setup.sh and run tests.sh executable chmod +x . - Validate: python validate.py --task {CVE-ID} . - Build: python build.py --task {CVE-ID} . This work was conducted as independent research. At the time of conducting the research and preparing this repository, I had no institutional affiliation. @misc{gattipinheiro2026cvebench, author = {Gatti Pinheiro, Giovanni}, title = {{CVE-Bench}: Benchmarking {LLM} Agents on Real-World Security Vulnerability Fixes}, year = {2026}, howpublished = {\url{https://giovannigatti.github.io/cve-bench}}, note = {Code available at \url{https://github.com/GiovanniGatti/cve-bench}} } MIT — see LICENSE /GiovanniGatti/cve-bench/blob/main/LICENSE .