This repository packages a few benchmarks and agents used by the Center for AI Standards and Innovation (CAISI) for cyber capability evaluations of AI systems. It aims to provide a standard, user-friendly interface to running agentic cyber benchmarks with the Inspect evaluation framework.
The package currently includes 2 agents and 3 benchmarks.
-
Based on CVE-Bench created by Yuxuan Zhu et al. .
-
CAISI's modifications are largely around the Inspect integration and automated evaluation.
-
This version of the benchmark has 7 tasks plus a dev tasks.
-
The upstream repository has 40 tasks.
-
This code is based off the public cybench fork maintained by the UK AI Security Institute here , originally developed by Andy Zhang et al.paper .
-
CAISI added original task descriptions, and updated system/task level prompts.
-
CAISI updated the benchmark to support running all samples within a single eval (i.e., produce one .eval file instead of 40).
-
After locally cloning a pwn.college dojo of tasks, this code can convert the tasks into Inspect-AI tasks.
-
The ctf-archive dojo provides 615 tasks.
-
CAISI developed the integration between pwn.college and Inspect to enable this. Note this repository only contains the integration, the pwn.college tasks live in their respective repos.
- Create a virtual environment in which to install ucb Install uv per its docs , then
uv venv
source .venv/bin/activate
- Install ucb:
uv sync
- Configure variables for your run.
You should now create a .env file and add the relevant API keys to it.
It is highly recommend to use a container registry via UCB_CONTAINER_REGISTRY which can
point at a domain and a directory prefix, e.g., "gitlab.com/caisi/cyber/ucb/". This should
end with a trailing slash.
ucb env-init # Populate .env from a template
vim .env # Then fill it in manually
- Build (or pull) containers
If you have previously pushed images to your container registry, you can skip building and simply
run ucb pull to download them.
Otherwise you'll need to build your images with ucb build. You can add a --push argument if you
want to push the images to a container registry, assuming you have set one up.
- Launch Ghidra-as-a-Service (Gaas) container
ucb gaas
- Confirm functionality by running a single task with a small budget:
inspect eval ucb/cybench --solver ucb/agent --model anthropic/claude-3-5-sonnet-20240620 --limit 1 --token-limit 2000
Ensure your GaaS server is running (e.g., with docker ps) if you haven't started it, launch it with ucb gaas.
GaaS will cache Ghidra projects and static analysis analysis results.
Once you've confirmed GaaS is running, you'll want to launch an eval with inspect eval or inspect eval-set
combined with the arguments described below.
You should specify at least one benchmark to run, either ucb/cybench or ucb/cvebench as part of your inspect command.
Note you can put both in the command and the tasks from both benchmarks will be run, but this is currently incompatible
with the UCB solvers.
You should also specify a single agent with --solver ucb/cybench_agent or --solver ucb/ctf_solver. You MUST specify an
agent.
- A model can be set with
--model=name, for example--model=openai/o3-mini-2025-01-31. If usingeval-set, multiple models can be specified and should be separated by a comma, for example--model=openai/o3-mini-2025-01-31,anthropic/claude-3-5-sonnet-20240620 - The total number of tokens can be limited with
--token-limit X. - Reasoning effort can be set with
--reasoning-effort <low/medium/high>for OpenAI reasoning models or--reasoning-tokens Xfor Anthropic reasoning models. - Limit the amount of resources used on your system with
--max-samples,--max-tasks,--max-sandboxes, and--max-subprocesses. - A number of epochs can be set with
--epochs, for example--epochs 5, default is 1. - For other arguments refer to the Inspect documentation .
You should specify an agent with --solver <name> where the name is either ucb/ctf_agent or ucb/cvebench_agent. The cvebench_agent
is simply a wrapper on the standard agent, paired with the task-specific graders for CVE bench tasks.
Agent behavior can be configured with the -S flag followed by a key-value pair with an equals sign, e.g., -S key=value.
sys_msg: str = CYBER_SYSTEM_MESSAGE: System prompt. Customized bycvebench_agent.use_ghidra_tool: bool = True: if the Ghidra-based tools should be provided to the agent. Disabled bycvebench_agent.max_submit_tool_calls: int = 10: Maximum number of calls to submit tool. -1 for unlimited.timeout int = 180: Timeout for tool calls.
Task level options can also be configured with the -T flag.
Task options vary depending on the benchmark you select, but the samples option is valid for all benchmarks:
samples: Optional[List[str]]: List of task names to include. Separate with commas (i.e.,-T samples=one,two). You can select samples by name across benchmarks. If unset, all tasks will be selected from the specified benchmarks.
These options will impact tasks from CVE-bench:
cve_details: bool = True: Should the CVE text be made available.target_details: bool = True: Should the agent be given a copy of relevant files from the target.writeup_details: bool = False: Should the text of a technical writeup be made available.dev_set: bool = False: Should only development tasks be selected?
These options will impact tasks from Cybench:
show_original_desc: bool = True: Should the original CTF task description be shown to the model (in addition to the standard task description).
inspect eval ucb/cvebench --model anthropic/claude-3-5-sonnet-20240620 --solver ucb/cvebench_agent --token-limit 100000
inspect eval ucb/cybench --model openai/o3-2025-04-16 -T samples=avatar,delulu --epochs 10 --solver ucb/ctf_agent --token-limit 150000
To run a large evaluation with high parallelism, we highly recommend using inspect eval-set, including the --no-fail-on-error flag, specifying
epochs, and setting max-tasks and max-sandboxes based on your system's resources. Note that eval-set needs a unique log-dir for each
evaluation you launch so it's typically important to set this as well instead of using the default logs. For example, an eval-set command
might look like:
inspect eval-set ucb/cvebench \
--solver ucb/cvebench_agent \
--model openai/o3-2025-04-16 \
--no-fail-on-error --epochs 4 \
--max-tasks=80 --max-sandboxes=140 \
--log-dir o3 \
--token-limit 2500000 \
--working-limit 7200
Spurious failures (e.g., docker containers that failed to start) can then be addressed by rerunning the same command after it finishes -
eval-set will parse the existing logs and attempt to finish previously-failed tasks.
src/ucb/containers/ contains generic containers used across tasks, for
example the container from which an agent will launch its attacks, and a
Ghidra-as-a-service (GaaS) container.
Benchmark information is stored in src/ucb/benchmarks/ where each benchmark
has its files stored in a unique subfolder (e.g., cybench). Within each benchmark,
a subfolder exists for each unique task (e.g., CTF problem) and individual samples
are specified in samples.yml. Typically there is one sample per task, but multiple
samples can be defined in a given samples.yml file (e.g., changing a task description
or the files provided to a model). Within the task folder (e.g.,
src/ucb/benchmarks/cybench/avatar) there are two required files, in addition to
any challenge-specific files (e.g. container files):
| File | Description |
|---|---|
samples.yml |
Details of the sample(s) (e.g. instructions and flag) |
compose.yml |
Docker Compose file which defines Docker infrastructure |
The glue to parse samples is stored in src/ucb/tasks/ which parses samples.yml files
and provides them to inspect.
Agents are implemented in src/ucb/agents.
- Create new directory within
src/ucb/samples/ - Add subfolders for each task, create required files, and task-specific containers
- Update
src/ucb/inspect/task.pyto support generating tasks for your benchmark.
This code can also be used with the pwncollege ctf-archive. To run with this benchmark, follow the setup instructions in docs/ctf_archive, then then run:
inspect eval ucb/ctf_archive \
-T root=/path/to/ctf-archive \
--solver ucb/ctf_archive_agent \
...
This repository is largely a combination of existing open source projects with some new glue between them. The authors of the upstream projects deserve significant credit for their effort!
- Creators of Inspect
- Creators of Inspect evals which contains a Cybench fork we drew inspiration from.
- Creators of Inspect cyber
Cybench authors. Paper. Repo. Thanks to Andy K. Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W. Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Jasper, Pura Peetathawatchai, Ari Glenn, Vikram Sivashankar, Daniel Zamoshchin, Leo Glikbarg, Derek Askaryar, Mike Yang, Teddy Zhang, Rishi Alluri, Nathan Tran, Rinnara Sangpisit, Polycarpos Yiorkadjis, Kenny Osele, Gautham Raghupathi, Dan Boneh, Daniel E. Ho, Percy Liang
@misc{
cybench
title={Cybench: A framework for evaluating cybersecurity capabilities and risks of language models},
author={Zhang, Andy K and Perry, Neil and Dulepet, Riya and Ji, Joey and Menders, Celeste and Lin, Justin W and Jones, Eliot and Hussein, Gashon and Liu, Samantha and Jasper, Donovan and others},
year={2024},
url={https://arxiv.org/abs/2408.08926}
}
CVE-Bench authors. Paper. Repo. Thanks to Yuxuan Zhu and Antony Kellermann and Dylan Bowman and Philip Li and Akul Gupta and Adarsh Danda and Richard Fang and Conner Jensen and Eric Ihli and Jason Benn and Jet Geronimo and Avi Dhir and Sudhit Rao and Kaicheng Yu and Twm Stone and Daniel Kang.
@misc{
cvebench,
title={CVE-Bench: A Benchmark for AI Agents’ Ability to Exploit Real-World Web Application Vulnerabilities},
author={Yuxuan Zhu and Antony Kellermann and Dylan Bowman and Philip Li and Akul Gupta and Adarsh Danda and Richard Fang and Conner Jensen and Eric Ihli and Jason Benn and Jet Geronimo and Avi Dhir and Sudhit Rao and Kaicheng Yu and Twm Stone and Daniel Kang},
year={2025},
url={https://arxiv.org/abs/2503.17332}
}
pwn.college creators. Paper and CTF-Archive maintainers.
@inproceedings{nelson2024pwn,
title={PWN The Learning Curve: Education-First CTF Challenges},
author={Nelson, Connor and Shoshitaishvili, Yan},
booktitle={Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1},
pages={937--943},
year={2024}
}