How do you measure the success with which different agents perform regular, real-world cybersecurity tasks?
Many companies already use agents within their development and deployment ecosystem, and attackers sometimes use swarms of offensive agents to attack thousands of companies at once. To help defenders prepare for this new era, we need a score that compares what these agents can do. Because of how important this is, labs in academia across the industry are starting to work on measuring it and setting a standard for how to do it correctly, so the topic is timely and increasingly relevant.
In this post I’ll share the key principles Wiz followed to tackle this benchmarking imperative in the relaunch of our Cyber Model Arena. The full methodology, how we graded, and gathered our tasks is available for download. As of the writing of this blog, the Wiz Cyber Model Arena v3.9 consists of 26 models, four general-purpose harnesses, 59 agents, 327 challenges, three attempts per challenge, and 1,000 seconds each (these numbers are constantly growing).
To create a benchmark, fix the benchmark objective first. Then build the tasks, environment, and the measurement around it. I’ll get into this and explain the principles that guide the process, in greater detail later in this post.
Key Takeaways
Choose the right tasks. Settle the why (objective), the what (data, ground truth and agent setup) and the how (metrics, evaluation function) before you write the first task. Build the data from the real distribution where you can, not from AI-generated or AI-labeled tasks.
Grade with code, not LLMs. Measure success directly and deterministically, and apply anti-cheat measures: isolate the agent from the server and the internet, and give it no way to look up answers. Do not rely on an LLM-as-a-judge unless it is well designed, calibrated and free of bias.
Optimize evaluation for realism. Evaluate the harness and model together, give the agent more than one try, and weigh success against what a solve costs.
Avoid common pitfalls. LLM judges, small samples, task imbalance, model-only evaluation and answer leakage each make a benchmark unreliable. And start small: one harness, a quick and cheap model, a few tasks and a simple grader. Scale only after it works.
Start with a problem statement
Most public cyber-agent benchmarks answer a different question than the one a defender is asking. A multiple-choice quiz benchmark measures model knowledge not production performance, and an LLM output judged by another model measures agreement between the two, not success. We aim to answer whether the agent can perform a real attack.
Write down the one question your benchmark must answer, and test every later decision against it. Ours:
Given only the task instruction, a general-purpose agent scaffold, a 1,000-second budget, and no external knowledge sources, does the agent produce a checkable effect on a real target?
Wiz Research's 2026 State of AI in the Cloud Report found that at least 57% of organizations have deployed self-hosted AI agent technology, and that MCP servers appear in at least 80% of observed cloud environments. People are already buying and defending against these systems. That is the why. The rest of this post is how to build a number that serves it.
Design your benchmark
Define three things before you create your first challenge.
Why. A defender or a buyer should be able to act on the number. "Higher pass@1" is not a why. "Can this agent, as shipped, exploit a live target from the instruction alone?" is.
What. Name the subject under evaluation. Here it is an agent (model + harness + tools + system prompt) run by an attacker, which is why we use realistic limits: 1,000 seconds, three attempts, a simple system prompt, and access to client-side code but not server-side code.
How. It’s important for the benchmark to report a single bottom line metric per example. To reduce many competing metrics to a single number, pick one optimizing metric and treat everything else as a satisficing constraint. If you have five competing metrics and no way to reduce them, which agent is better becomes a debate over the metrics.
Make the tasks match the job
A benchmark can only speak for the tasks it contains. Sample only the hard tasks and you are no longer measuring success on the job, you are measuring success on hard tasks. And a benchmark that claims to measure "offensive capability" from a single task category oversells itself.
So we built four categories of equal weight from sources that already exist in the wild. We did not generate or label tasks with an LLM.
Category
Tasks
What it tests
Code vulnerabilities
221
Finding known vulnerabilities in stripped open-source packages, or exploiting a live 1-day CVE
API security
54
Attacking black-box APIs with no source code
Web security
29
Exploiting real PHP web apps with the client source visible
Cloud security
23
Exploiting misconfigurations across AWS, Azure, GCP and Kubernetes-style setups
The agents behave very differently from one category to the next. Averaged over the 59 agents, pass@1 is 78.0% on API security, 64.6% on web, 56.5% on cloud and 38.2% on code vulnerabilities. The top overall agent scores 74.7% on web and 65.8% on code (while web tops out at 90.8%). Another agent scores 9.7% on code and still 71.6% on API.
Size matters too. Many recent cybersecurity benchmarks have fewer than 30 tasks. At a 50% success rate, 30 tasks give a 95% confidence interval of roughly ±18 points, so an agent that scores 15 points higher than another is not necessarily better.
Grade an effect, not with LLMs
Ground truth is the foundation of evaluation. It has been standard practice to grade with an LLM-as-a-Judge, but judges do not directly measure success. They are biased (they often prefer the first option), random (they give different results each run), expensive, and slow, and they must be calibrated and de-biased before you can use them in a long-lived benchmark. When you use a judge, you measure whether an LLM thinks the agent succeeded. That difference is a debt baked into the system.
We use code instead. Every challenge is designed to leave a checkable effect behind or report precise structured outputs, and a deterministic grader inspects it after the agent is gone. Code graders are cheap, re-runnable, and not a model you have to calibrate.
Anti-cheating via isolation
Agents will shortcut a task when they can. They look the answer up online, read a solution off disk, or manipulate the grader. A score collected with web search, a CVE database or the challenge name in the folder measures something else: success at the task or success at breaking it. CTFusion moved their solve rate by as much as 12 points just by raising web-search access.
As for contamination, we cannot prove our agents never saw a task in pretraining. We can prove they cannot fetch one during the trial. Anti-cheat belongs in the environment, not in the prompt.
Start small
Building a benchmark is a nontrivial, evolving effort, and it is tempting to write 300 challenges before anything runs end to end. We apply lean-startup rules instead: get a small end-to-end baseline you can inspect by hand, then iterate.
You need ten tasks you can re-run, one harness you can deploy, and an evaluation metric a grader can check. After you’re convinced it works, you can scale it up.
What we found
Pair the harness with the model
A model on its own is a text predictor. It becomes an attacker inside a harness that runs commands and applies tools, and models do better or worse in different harnesses. Across the 22 models we ran on both Claude Code and ADK React, the mean absolute gap was 5.1 pass@1 points. Claude models prefer Claude Code, and Gemini 3.8 Flash Cyber prefers the thinner ADK loop.
The conclusion: test the system as a whole. Either pair the harness to the model (and as we’ve seen it’s best to match the harness to the model's creator), or fix one general harness and compare models inside it.
Attackers get more than one try
A real attacker does not stop after one failure. We gave every agent three attempts per challenge, and the extra attempts did more than lift scores. For some agents they even changed which one comes out ahead.
Accuracy has a price
Give an agent more time and money and it will usually solve more. So the useful chart plots success against what a solve costs. Any agent off the Pareto frontier is strictly worse than one on it. For example, Claude Code with GPT-5.6 Sol averages 64.2% pass@1 at $8.29 and 27.2 minutes per solve, however the Pareto frontier Codex + GPT-6 Luna is strictly better, averaging 64.5% pass@1 at only $0.05 and 6.3 minutes per solve.
Keep a short list of agents on the frontier and ignore the rest. That lets you ask "for this budget, how do I maximize success?" or "for this success rate, how do I minimize cost?"
Limitations
Four simple harnesses. A cyber-specialized harness would likely score higher, but we chose to measure a simple attacker.
String equality on code vulnerabilities. A semantically correct root cause that misses the expected symbol scores 0. We chose precision over recall.
Public web tasks. websec.fr is a known corpus. Isolation stops the agent from looking up the answer, not from remembering it from training (possibly resulting in benchmark contamination). A time cap. 1,000 seconds is enough for most successful attacks we observed. In the future we think it is better to limit cost than time.
Confidence intervals. We have yet to publish them on the website, and we are working on it.
Conclusion
Our Cyber Model Arena shows how different agents perform on common cybersecurity tasks, across 327 isolated challenges in four categories. To build an agentic cybersecurity benchmark, fix the why, the what and the how first, and define the agent, the task mix, the grader, the isolation, the budget and the attempts. Get one wrong and the benchmark becomes garbage in, garbage out.
Appendix: Results for All 59 Agents
Every agent, ordered by pass@1. Scores are percentages.
A threat actor used IAM credentials from Qubole's AWS account to assume integration roles in customer AWS environments. Here's what we observed and how to hunt for it.