ExploitGym AI benchmark source code

ExploitGym is a large-scale, realistic benchmark built from real-world vulnerabilities across userspace programs, Google's V8 engine, and the Linux kernel, designed to evaluate AI agents' ability to develop exploits.

uv sync --extra proxy

bash scripts/setup/setup_data.sh

bash scripts/setup/validate.sh

docker pull ubuntu/squid:latest

uv run scripts/setup/pull_images.py data/task_ids/sample.txt

export OPENAI_API_KEY=...
export ANTHROPIC_API_KEY=...
uv run scripts/setup/pre_run.py data/task_ids/sample.txt

export CYBERGYM_ADMIN_KEY=...
uv run examples/run_agent.py --help

Detailed setup steps (system dependencies, GDB, static node, agent CLIs) live in docs/setup.md.

Setup: Python deps, GDB, socat/nc, node + agent CLIsDocker images: pulling target images per task familyEvaluation: controller / firewall / LLM proxy +examples/run_agent.py

Defenses: disabling system defenses (ASLR, etc.)Firewall: outbound network isolation for agent containersSubmission: submission format and requirements for the benchmark leaderboard

The released benchmark is actively maintained. The current release is v1.0 with 869 instances. See CHANGELOG.md for the full version history. The canonical task list for the current release is data/task_ids/v1.txt

.

If you use ExploitGym in your research, please cite:

@article{wang2026exploitgym,
  title={ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?},
  author={Wang, Zhun and Schiller, Nico and Li, Hongwei and Sesha Narayana, Srijiith and Nasr, Milad and Carlini, Nicholas and Qi, Xiangyu and Wallace, Eric and Bursztein, Elie and Invernizzi, Luca and Thomas, Kurt and Shoshitaishvili, Yan and Guo, Wenbo and He, Jingxuan and Holz, Thorsten and Song, Dawn},
  journal={arXiv preprint arXiv:2605.11086},
  year={2026}
}

The source code is licensed under Apache-2.0. The bundled task data under data/tasks/

derives from external upstreams and retains their respective licenses, see DATA_LICENSE.md.