{"slug": "ask-hn-which-cves-should-i-add-to-my-python-security-benchmark-for-ai-agents", "title": "Ask HN: Which CVEs should I add to my Python security benchmark for AI agents?", "summary": "A new open-source benchmark called CVE-Bench evaluates large language model agents on fixing real-world security vulnerabilities by running them inside sandboxed Docker containers and scoring them against maintainer security test suites. The benchmark supports Python 3.12+, Docker, and models from OpenAI, Anthropic, and Poolside, with each task containing a vulnerable SHA, fixed SHA, and multiple prompt types including advisory, diagnose, and locate. The project includes build, validation, and benchmarking scripts that produce JSON result files for each model and prompt combination.", "body_md": "A benchmark for evaluating LLM agents on fixing real-world security vulnerabilities. Agents run inside sandboxed Docker containers and are scored against the maintainer's security test suite.\n\n- Python 3.12+\n- Docker\n`OPENAI_API_KEY`\n\n,`ANTHROPIC_API_KEY`\n\n, and/or`POOLSIDE_API_KEY`\n\nin your environment (or a`.env`\n\nfile)\n\nInstall dependencies:\n\n```\npip install poetry\npoetry install\n```\n\nEach task lives under `tasks/{CVE-ID}/`\n\nand contains:\n\n```\ntasks/CVE-2026-33175/\n├── meta.json           # GHSA ID, CWE, CVSS, repo URL, vulnerable and fixed SHAs\n├── setup.sh            # Clones repo, checks out the vulnerable SHA, installs dependencies\n├── run_tests.sh        # Injects test_security.py into the repo and runs pytest\n├── test_security.py    # Security tests (xfail on vulnerable code, pass on the fix)\n├── advisory.md         # Full GHSA advisory (richest prompt)\n├── diagnose.md         # Behavioural description only — no file or function names\n├── locate.md           # File and function only — no description of the flaw\n└── Dockerfile          # Optional; only present when the task needs extra system deps\n```\n\n`meta.json`\n\nexample:\n\n```\n{\n  \"ghsa_id\": \"GHSA-xxxx-xxxx-xxxx\",\n  \"cwe\": [\"CWE-287\"],\n  \"cvss\": 9.1,\n  \"repo\": {\n    \"url\": \"https://github.com/org/project\",\n    \"vulnerable_sha\": \"abc123^\",\n    \"fixed_sha\": \"abc123\"\n  }\n}\n```\n\n`setup.sh`\n\nis idempotent and safe to re-run. `test_security.py`\n\nis kept hidden from the agent during the run and injected only after the agent finishes.\n\n```\npython build.py\n```\n\nThis builds:\n\n- A shared base image (\n`cve-bench/base`\n\n) — Python 3.12, git, poetry, and the harness. - One task image per task (\n`cve-bench/{task-id}`\n\n) — extends the base, copies the task directory, and runs`setup.sh`\n\n.\n\n**Options:**\n\n```\n# Build specific tasks only\npython build.py --task CVE-2026-33175 CVE-2026-42561\n\n# Skip rebuilding the base image\npython build.py --skip-base\n```\n\nTask images are built in parallel (up to 5 workers). If a task directory contains a `Dockerfile`\n\n, it is used instead of the generic `docker/task.Dockerfile`\n\n.\n\nBefore running the benchmark, verify that each task's security tests correctly distinguish vulnerable from fixed code:\n\n```\npython validate.py\n```\n\nFor each task, this runs three phases inside the task container:\n\n| Phase | What it checks |\n|---|---|\nvulnerable |\nSecurity tests must fail (or xfail) on the vulnerable SHA |\nfixed |\nSecurity tests must pass on the fixed SHA |\nregression |\nNon-security tests must pass on the fixed SHA |\n\nResults are displayed as a live table. Exit code is 1 if any task fails any phase.\n\n```\n# Validate specific tasks only\npython validate.py --task CVE-2026-33175 GHSA-r758-8hxw-4845\n\n# Skip rebuilding images before validation\npython validate.py --skip-build\npython benchmark.py --model openai:gpt-5.5 poolside:laguna-m.1 --prompt-type advisory\n```\n\n**Options:**\n\n| Flag | Description | Default |\n|---|---|---|\n`--model` |\nOne or more `provider:model-id` strings |\nall configured models |\n`--prompt-type` |\n`advisory` , `diagnose` , `locate` , or any combination |\nall three |\n`--task` |\nOne or more task IDs | all tasks |\n`--clean` |\nDelete existing results for the selected scope before starting | off |\n\n**Supported providers:**\n\n| Provider | Format | API key env var |\n|---|---|---|\n| OpenAI | `openai:gpt-5.5` |\n`OPENAI_API_KEY` |\n| Anthropic | `anthropic:claude-haiku-4-5-20251001` |\n`ANTHROPIC_API_KEY` |\n| Poolside | `poolside:laguna-m.1` |\n`POOLSIDE_API_KEY` |\n\nEach run produces a JSON result file in `results/`\n\n:\n\n```\nresults/{task-id}__{provider}:{model}__{prompt-type}.json\n```\n\nExisting result files are skipped automatically. Runs execute concurrently across tasks (up to 20 workers), with per-provider rate limiting (one active request per provider at a time) to avoid 429s.\n\nEach result file is a JSON object with the following structure:\n\n```\n{\n  \"cve_id\": \"CVE-2026-33175\",\n  \"model_id\": \"openai:gpt-5.5\",\n  \"prompt_type\": \"advisory\",\n  \"timestamp\": \"2026-05-01T12:00:00\",\n  \"model_duration_s\": 142.3,\n  \"test_duration_s\": 8.1,\n  \"turns\": [\n    {\n      \"tool_calls_and_results\": [...],\n      \"input_tokens\": 12400,\n      \"output_tokens\": 310\n    }\n  ],\n  \"tests\": [\n    {\n      \"kind\": \"security\",\n      \"name\": \"test_email_verified\",\n      \"outcome\": \"passed\"\n    }\n  ]\n}\n```\n\n`tests[].kind`\n\nis either `\"security\"`\n\n(from `test_security.py`\n\n) or `\"regression\"`\n\n(from the project's own test suite). A run is considered solved only if all security tests pass and no regression tests fail.\n\n```\npython generate_charts.py\n```\n\nReads all result files from `results/`\n\nand writes SVG charts to `docs/images/charts/`\n\n. Requires Chrome/Chromium for Bokeh's headless export (via `chromedriver-binary`\n\n).\n\nThe harness runs inside each Docker container as `python -m harness.run`\n\n. It is responsible for loading the prompt, running the agentic loop, and writing the result file.\n\n```\nsrc/harness/\n├── run.py                  # Entry point; parses args, wires components, calls BenchmarkRunner\n├── client/\n│   ├── factory.py          # Parses provider:model-id, returns the correct LLMClient\n│   ├── _client.py          # Abstract LLMClient, ToolCall and LLMTurn dataclasses\n│   ├── anthropic.py        # Anthropic SDK integration\n│   └── oai.py              # OpenAI SDK integration (also used for Poolside)\n├── agent/\n│   ├── core.py             # Agentic loop: calls client, dispatches tool calls, threads messages\n│   └── runner.py           # Wraps Agent, tracks timing and turn list\n├── bench/\n│   ├── runner.py           # Orchestrates setup → agent → security tests → regression tests\n│   ├── result.py           # BenchmarkResult and TestResult dataclasses, JSON serialisation\n│   └── repository.py       # Writes result files to disk\n└── task/\n    ├── tools.py             # Tool implementations: ListFiles, ReadFile, SearchInFiles,\n    │                        #   EditFile, CreateFile, DeleteFile, RunPytest\n    └── prompt_loader.py     # Reads advisory.md / diagnose.md / locate.md\n```\n\n**Tools available to the agent:**\n\n| Tool | Description |\n|---|---|\n`list_files` |\nList files and directories in the repository |\n`read_file` |\nRead file contents, optionally a line range |\n`search_in_files` |\nRegex search across the codebase with optional file glob |\n`edit_file` |\nReplace a range of lines in an existing file |\n`create_file` |\nCreate a new file |\n`delete_file` |\nDelete a file |\n`run_pytest` |\nRun the project's test suite; returns a JSON report |\n\nAll tools validate paths against the repository root to prevent directory traversal. The agent does not have access to `test_security.py`\n\nor to the git history.\n\nThe agent loop runs for at most 20 turns. If the turn ceiling is reached, the run is recorded as-is and the security tests are still executed against whatever state the agent left the repository in.\n\n- Create\n`tasks/{CVE-ID}/`\n\nand add`meta.json`\n\n,`setup.sh`\n\n,`run_tests.sh`\n\n,`test_security.py`\n\n,`advisory.md`\n\n,`diagnose.md`\n\n,`locate.md`\n\n. - Make\n`setup.sh`\n\nand`run_tests.sh`\n\nexecutable (`chmod +x`\n\n). - Validate:\n`python validate.py --task {CVE-ID}`\n\n. - Build:\n`python build.py --task {CVE-ID}`\n\n.\n\nThis work was conducted as independent research. At the time of conducting the research and preparing this repository, I had no institutional affiliation.\n\n```\n@misc{gattipinheiro2026cvebench,\n  author       = {Gatti Pinheiro, Giovanni},\n  title        = {{CVE-Bench}: Benchmarking {LLM} Agents on Real-World Security Vulnerability Fixes},\n  year         = {2026},\n  howpublished = {\\url{https://giovannigatti.github.io/cve-bench}},\n  note         = {Code available at \\url{https://github.com/GiovanniGatti/cve-bench}}\n}\n```\n\nMIT — see [LICENSE](/GiovanniGatti/cve-bench/blob/main/LICENSE).", "url": "https://wpnews.pro/news/ask-hn-which-cves-should-i-add-to-my-python-security-benchmark-for-ai-agents", "canonical_source": "https://github.com/GiovanniGatti/cve-bench", "published_at": "2026-07-23 14:10:28+00:00", "updated_at": "2026-07-23 14:22:49.268155+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-research", "ai-tools", "developer-tools"], "entities": ["OpenAI", "Anthropic", "Poolside", "GitHub", "Docker", "Python", "CVE-Bench"], "alternates": {"html": "https://wpnews.pro/news/ask-hn-which-cves-should-i-add-to-my-python-security-benchmark-for-ai-agents", "markdown": "https://wpnews.pro/news/ask-hn-which-cves-should-i-add-to-my-python-security-benchmark-for-ai-agents.md", "text": "https://wpnews.pro/news/ask-hn-which-cves-should-i-add-to-my-python-security-benchmark-for-ai-agents.txt", "jsonld": "https://wpnews.pro/news/ask-hn-which-cves-should-i-add-to-my-python-security-benchmark-for-ai-agents.jsonld"}}