ExploitGym AI benchmark source code

Researchers from UC Berkeley and Google released ExploitGym v1.0, a benchmark with 869 real-world vulnerability instances across userspace programs, Google's V8 engine, and the Linux kernel, designed to evaluate AI agents' ability to develop exploits. The open-source benchmark, led by Zhun Wang and colleagues, aims to measure whether AI can turn security vulnerabilities into real attacks.

ExploitGym is a large-scale, realistic benchmark built from real-world vulnerabilities across userspace programs, Google's V8 engine, and the Linux kernel, designed to evaluate AI agents' ability to develop exploits. 1. Python deps uv sync --extra proxy 2. Build runtime artifacts gdb, socat, nc, node + agent CLIs and extract task data bash scripts/setup/setup data.sh 3. Verify the install bash scripts/setup/validate.sh 4. Pull the Firewall Squid image docker pull ubuntu/squid:latest 5. Pull the Docker images for the tasks you want to run uv run scripts/setup/pull images.py data/task ids/sample.txt 6. Start the controller, firewall, and LLM proxy. pre run.py runs the readiness checks and starts all three auto-detecting any already running , or start them by hand — see docs/eval.md export OPENAI API KEY=... export ANTHROPIC API KEY=... uv run scripts/setup/pre run.py data/task ids/sample.txt 7. Run the agent export CYBERGYM ADMIN KEY=... uv run examples/run agent.py --help Detailed setup steps system dependencies, GDB, static node, agent CLIs live in docs/setup.md /sunblaze-ucb/exploitgym/blob/main/docs/setup.md . Setup /sunblaze-ucb/exploitgym/blob/main/docs/setup.md : Python deps, GDB, socat/nc, node + agent CLIs Docker images /sunblaze-ucb/exploitgym/blob/main/docs/docker images.md : pulling target images per task family Evaluation /sunblaze-ucb/exploitgym/blob/main/docs/eval.md : controller / firewall / LLM proxy + examples/run agent.py Defenses /sunblaze-ucb/exploitgym/blob/main/docs/defenses.md : disabling system defenses ASLR, etc. Firewall /sunblaze-ucb/exploitgym/blob/main/docs/firewall.md : outbound network isolation for agent containers Submission /sunblaze-ucb/exploitgym/blob/main/docs/submission.md : submission format and requirements for the benchmark leaderboard The released benchmark is actively maintained. The current release is v1.0 with 869 instances. See CHANGELOG.md /sunblaze-ucb/exploitgym/blob/main/CHANGELOG.md for the full version history. The canonical task list for the current release is data/task ids/v1.txt . If you use ExploitGym in your research, please cite: @article{wang2026exploitgym, title={ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?}, author={Wang, Zhun and Schiller, Nico and Li, Hongwei and Sesha Narayana, Srijiith and Nasr, Milad and Carlini, Nicholas and Qi, Xiangyu and Wallace, Eric and Bursztein, Elie and Invernizzi, Luca and Thomas, Kurt and Shoshitaishvili, Yan and Guo, Wenbo and He, Jingxuan and Holz, Thorsten and Song, Dawn}, journal={arXiv preprint arXiv:2605.11086}, year={2026} } The source code is licensed under Apache-2.0 /sunblaze-ucb/exploitgym/blob/main/LICENSE . The bundled task data under data/tasks/ derives from external upstreams and retains their respective licenses, see DATA LICENSE.md /sunblaze-ucb/exploitgym/blob/main/DATA LICENSE.md .