Caisi Cyber Evaluations

The Center for AI Standards and Innovation (CAISI) released a repository packaging 2 agents and 3 benchmarks for agentic cyber capability evaluations of AI systems, built on the Inspect evaluation framework. The package includes a CVE-Bench version with 7 tasks plus a dev task (down from 40 tasks in the upstream repository), a Cybench fork with original task descriptions and single-eval support replacing 40 separate .eval files, and an integration that converts pwn.college dojo tasks, including the 615-task ctf-archive dojo, into Inspect-AI tasks. The tooling requires a Ghidra-as-a-Service container and supports models such as anthropic/claude-3-5-sonnet-20240620 and openai/o3-mini-2025-01-31 with configurable token limits and reasoning effort.

This repository packages a few benchmarks and agents used by the Center for AI Standards and Innovation CAISI https://www.nist.gov/caisi for cyber capability evaluations of AI systems. It aims to provide a standard, user-friendly interface to running agentic cyber benchmarks with the Inspect evaluation framework https://github.com/UKGovernmentBEIS/inspect ai/ . The package currently includes 2 agents and 3 benchmarks. - Based on CVE-Bench created by Yuxuan Zhu et al. https://arxiv.org/pdf/2503.17332 . - CAISI's modifications are largely around the Inspect integration and automated evaluation. - This version of the benchmark has 7 tasks plus a dev tasks. - The upstream repository https://github.com/uiuc-kang-lab/cve-bench has 40 tasks. - This code is based off the public cybench fork maintained by the UK AI Security Institute here https://github.com/UKGovernmentBEIS/inspect evals/tree/main/src/inspect evals/cybench , originally developed by Andy Zhang et al. paper https://arxiv.org/abs/2408.08926 . - CAISI added original task descriptions, and updated system/task level prompts. - CAISI updated the benchmark to support running all samples within a single eval i.e., produce one .eval file instead of 40 . - After locally cloning a pwn.college dojo of tasks, this code can convert the tasks into Inspect-AI tasks. - The ctf-archive dojo https://github.com/pwncollege/ctf-archive/ provides 615 tasks. - CAISI developed the integration between pwn.college and Inspect to enable this. Note this repository only contains the integration, the pwn.college tasks live in their respective repos. 1. Create a virtual environment in which to install ucb Install uv per its docs https://docs.astral.sh/uv/getting-started/installation/ , then uv venv source .venv/bin/activate 1. Install ucb: uv sync 1. Configure variables for your run. You should now create a .env file and add the relevant API keys to it. It is highly recommend to use a container registry via UCB CONTAINER REGISTRY which can point at a domain and a directory prefix, e.g., "gitlab.com/caisi/cyber/ucb/". This should end with a trailing slash. ucb env-init Populate .env from a template vim .env Then fill it in manually 1. Build or pull containers If you have previously pushed images to your container registry, you can skip building and simply run ucb pull to download them. Otherwise you'll need to build your images with ucb build . You can add a --push argument if you want to push the images to a container registry, assuming you have set one up. 1. Launch Ghidra-as-a-Service Gaas container ucb gaas 1. Confirm functionality by running a single task with a small budget: inspect eval ucb/cybench --solver ucb/agent --model anthropic/claude-3-5-sonnet-20240620 --limit 1 --token-limit 2000 Ensure your GaaS server is running e.g., with docker ps if you haven't started it, launch it with ucb gaas . GaaS will cache Ghidra projects and static analysis analysis results. Once you've confirmed GaaS is running, you'll want to launch an eval with inspect eval or inspect eval-set combined with the arguments described below. You should specify at least one benchmark to run, either ucb/cybench or ucb/cvebench as part of your inspect command. Note you can put both in the command and the tasks from both benchmarks will be run, but this is currently incompatible with the UCB solvers. You should also specify a single agent with --solver ucb/cybench agent or --solver ucb/ctf solver . You MUST specify an agent. - A model can be set with --model=name , for example --model=openai/o3-mini-2025-01-31 . If using eval-set , multiple models can be specified and should be separated by a comma, for example --model=openai/o3-mini-2025-01-31,anthropic/claude-3-5-sonnet-20240620 - The total number of tokens can be limited with --token-limit X . - Reasoning effort can be set with --reasoning-effort <low/medium/high for OpenAI reasoning models or --reasoning-tokens X for Anthropic reasoning models. - Limit the amount of resources used on your system with --max-samples , --max-tasks , --max-sandboxes , and --max-subprocesses . - A number of epochs can be set with --epochs , for example --epochs 5 , default is 1. - For other arguments refer to the Inspect documentation https://inspect.ai-safety-institute.org.uk/reference/inspect ai.html eval . You should specify an agent with --solver <name where the name is either ucb/ctf agent or ucb/cvebench agent . The cvebench agent is simply a wrapper on the standard agent, paired with the task-specific graders for CVE bench tasks. Agent behavior can be configured with the -S flag followed by a key-value pair with an equals sign, e.g., -S key=value . - sys msg: str = CYBER SYSTEM MESSAGE : System prompt. Customized by cvebench agent . - use ghidra tool: bool = True : if the Ghidra-based tools should be provided to the agent. Disabled by cvebench agent . - max submit tool calls: int = 10 : Maximum number of calls to submit tool. -1 for unlimited. - timeout int = 180 : Timeout for tool calls. Task level options can also be configured with the -T flag. Task options vary depending on the benchmark you select, but the samples option is valid for all benchmarks: - samples: Optional List str : List of task names to include. Separate with commas i.e., -T samples=one,two . You can select samples by name across benchmarks. If unset, all tasks will be selected from the specified benchmarks. These options will impact tasks from CVE-bench: - cve details: bool = True : Should the CVE text be made available. - target details: bool = True : Should the agent be given a copy of relevant files from the target. - writeup details: bool = False : Should the text of a technical writeup be made available. - dev set: bool = False : Should only development tasks be selected? These options will impact tasks from Cybench: - show original desc: bool = True : Should the original CTF task description be shown to the model in addition to the standard task description . inspect eval ucb/cvebench --model anthropic/claude-3-5-sonnet-20240620 --solver ucb/cvebench agent --token-limit 100000 inspect eval ucb/cybench --model openai/o3-2025-04-16 -T samples=avatar,delulu --epochs 10 --solver ucb/ctf agent --token-limit 150000 To run a large evaluation with high parallelism, we highly recommend using inspect eval-set , including the --no-fail-on-error flag, specifying epochs, and setting max-tasks and max-sandboxes based on your system's resources. Note that eval-set needs a unique log-dir for each evaluation you launch so it's typically important to set this as well instead of using the default logs . For example, an eval-set command might look like: inspect eval-set ucb/cvebench \ --solver ucb/cvebench agent \ --model openai/o3-2025-04-16 \ --no-fail-on-error --epochs 4 \ --max-tasks=80 --max-sandboxes=140 \ --log-dir o3 \ --token-limit 2500000 \ --working-limit 7200 Spurious failures e.g., docker containers that failed to start can then be addressed by rerunning the same command after it finishes - eval-set will parse the existing logs and attempt to finish previously-failed tasks. src/ucb/containers/ contains generic containers used across tasks, for example the container from which an agent will launch its attacks, and a Ghidra-as-a-service GaaS container. Benchmark information is stored in src/ucb/benchmarks/ where each benchmark has its files stored in a unique subfolder e.g., cybench . Within each benchmark, a subfolder exists for each unique task e.g., CTF problem and individual samples are specified in samples.yml . Typically there is one sample per task, but multiple samples can be defined in a given samples.yml file e.g., changing a task description or the files provided to a model . Within the task folder e.g., src/ucb/benchmarks/cybench/avatar there are two required files, in addition to any challenge-specific files e.g. container files : | File | Description | |---|---| | samples.yml | Details of the sample s e.g. instructions and flag | | compose.yml | Docker Compose file which defines Docker infrastructure | The glue to parse samples is stored in src/ucb/tasks/ which parses samples.yml files and provides them to inspect. Agents are implemented in src/ucb/agents . 1. Create new directory within src/ucb/samples/ 2. Add subfolders for each task, create required files, and task-specific containers 3. Update src/ucb/inspect/task.py to support generating tasks for your benchmark. This code can also be used with the pwncollege ctf-archive https://github.com/pwncollege/ctf-archive/ . To run with this benchmark, follow the setup instructions in docs/ctf archive https://github.com/usnistgov/caisi-cyber-evals/blob/main/docs/ctf archive.md , then then run: inspect eval ucb/ctf archive \ -T root=/path/to/ctf-archive \ --solver ucb/ctf archive agent \ ... This repository is largely a combination of existing open source projects with some new glue between them. The authors of the upstream projects deserve significant credit for their effort - Creators of Inspect https://github.com/UKGovernmentBEIS/inspect ai/ - Creators of Inspect evals https://github.com/UKGovernmentBEIS/inspect evals/ which contains a Cybench fork we drew inspiration from. - Creators of Inspect cyber https://github.com/UKGovernmentBEIS/inspect cyber Cybench authors. Paper https://arxiv.org/pdf/2408.08926 . Repo https://github.com/andyzorigin/cybench . Thanks to Andy K. Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W. Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Jasper, Pura Peetathawatchai, Ari Glenn, Vikram Sivashankar, Daniel Zamoshchin, Leo Glikbarg, Derek Askaryar, Mike Yang, Teddy Zhang, Rishi Alluri, Nathan Tran, Rinnara Sangpisit, Polycarpos Yiorkadjis, Kenny Osele, Gautham Raghupathi, Dan Boneh, Daniel E. Ho, Percy Liang @misc{ cybench title={Cybench: A framework for evaluating cybersecurity capabilities and risks of language models}, author={Zhang, Andy K and Perry, Neil and Dulepet, Riya and Ji, Joey and Menders, Celeste and Lin, Justin W and Jones, Eliot and Hussein, Gashon and Liu, Samantha and Jasper, Donovan and others}, year={2024}, url={https://arxiv.org/abs/2408.08926} } CVE-Bench authors. Paper https://arxiv.org/abs/2503.17332 . Repo https://github.com/uiuc-kang-lab/cve-bench/blob/main/README.md . Thanks to Yuxuan Zhu and Antony Kellermann and Dylan Bowman and Philip Li and Akul Gupta and Adarsh Danda and Richard Fang and Conner Jensen and Eric Ihli and Jason Benn and Jet Geronimo and Avi Dhir and Sudhit Rao and Kaicheng Yu and Twm Stone and Daniel Kang. @misc{ cvebench, title={CVE-Bench: A Benchmark for AI Agents’ Ability to Exploit Real-World Web Application Vulnerabilities}, author={Yuxuan Zhu and Antony Kellermann and Dylan Bowman and Philip Li and Akul Gupta and Adarsh Danda and Richard Fang and Conner Jensen and Eric Ihli and Jason Benn and Jet Geronimo and Avi Dhir and Sudhit Rao and Kaicheng Yu and Twm Stone and Daniel Kang}, year={2025}, url={https://arxiv.org/abs/2503.17332} } pwn.college https://pwn.college creators. Paper https://dl.acm.org/doi/abs/10.1145/3626252.3630912 and CTF-Archive https://github.com/pwncollege/ctf-archive maintainers. @inproceedings{nelson2024pwn, title={PWN The Learning Curve: Education-First CTF Challenges}, author={Nelson, Connor and Shoshitaishvili, Yan}, booktitle={Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1}, pages={937--943}, year={2024} }