{"slug": "how-to-evaluate-ai-agent-cybersecurity-capabilities", "title": "How to Evaluate AI Agent Cybersecurity Capabilities", "summary": "Wiz Research relaunched its Cyber Model Arena as v3.9, a benchmark of 26 models, four general-purpose harnesses, 59 agents and 327 challenges, with three attempts per challenge and a 1,000-second budget each, to measure whether an agent given only a task instruction can produce a checkable effect on a real target. Wiz Research's 2026 State of AI in the Cloud Report found at least 57% of organizations have deployed self-hosted AI agent technology and MCP servers appear in at least 80% of observed cloud environments. Wiz's methodology calls for grading with deterministic code rather than LLM judges, isolating agents from servers and the internet to prevent answer leakage, and evaluating the harness and model together.", "body_md": "How do you measure the success with which different agents perform regular, real-world cybersecurity tasks?\n\nMany companies already use agents within their development and deployment ecosystem, and attackers sometimes use swarms of offensive agents to attack thousands of companies at once. To help defenders prepare for this new era, we need a score that compares what these agents can do. Because of how important this is, labs in academia across the industry are starting to work on measuring it and setting a standard for how to do it correctly, so the topic is timely and increasingly relevant.\n\nIn this post I’ll share the key principles Wiz followed to tackle this benchmarking imperative in the relaunch of our Cyber Model Arena. The full methodology, how we graded, and gathered our tasks is available for download. As of the writing of this blog, the Wiz Cyber Model Arena v3.9 consists of 26 models, four general-purpose harnesses, 59 agents, 327 challenges, three attempts per challenge, and 1,000 seconds each (these numbers are constantly growing).\n\nTo create a benchmark, fix the benchmark objective first. Then build the tasks, environment, and the measurement around it. I’ll get into this and explain the principles that guide the process, in greater detail later in this post.\n\nKey Takeaways\n\nChoose the right tasks. Settle the why (objective), the what (data, ground truth and agent setup) and the how (metrics, evaluation function) before you write the first task. Build the data from the real distribution where you can, not from AI-generated or AI-labeled tasks.\n\nGrade with code, not LLMs. Measure success directly and deterministically, and apply anti-cheat measures: isolate the agent from the server and the internet, and give it no way to look up answers. Do not rely on an LLM-as-a-judge unless it is well designed, calibrated and free of bias.\n\nOptimize evaluation for realism. Evaluate the harness and model together, give the agent more than one try, and weigh success against what a solve costs.\n\nAvoid common pitfalls. LLM judges, small samples, task imbalance, model-only evaluation and answer leakage each make a benchmark unreliable. And start small: one harness, a quick and cheap model, a few tasks and a simple grader. Scale only after it works.\n\nStart with a problem statement\n\nMost public cyber-agent benchmarks answer a different question than the one a defender is asking. A multiple-choice quiz benchmark measures model knowledge not production performance, and an LLM output judged by another model measures agreement between the two, not success. We aim to answer whether the agent can perform a real attack.\n\nWrite down the one question your benchmark must answer, and test every later decision against it. Ours:\n\nGiven only the task instruction, a general-purpose agent scaffold, a 1,000-second budget, and no external knowledge sources, does the agent produce a checkable effect on a real target?\n\nWiz Research's 2026 State of AI in the Cloud Report found that at least 57% of organizations have deployed self-hosted AI agent technology, and that MCP servers appear in at least 80% of observed cloud environments. People are already buying and defending against these systems. That is the why. The rest of this post is how to build a number that serves it.\n\nDesign your benchmark\n\nDefine three things before you create your first challenge.\n\nWhy. A defender or a buyer should be able to act on the number. \"Higher pass@1\" is not a why. \"Can this agent, as shipped, exploit a live target from the instruction alone?\" is.\n\nWhat. Name the subject under evaluation. Here it is an agent (model + harness + tools + system prompt) run by an attacker, which is why we use realistic limits: 1,000 seconds, three attempts, a simple system prompt, and access to client-side code but not server-side code.\n\nHow. It’s important for the benchmark to report a single bottom line metric per example. To reduce many competing metrics to a single number, pick one optimizing metric and treat everything else as a satisficing constraint. If you have five competing metrics and no way to reduce them, which agent is better becomes a debate over the metrics.\n\nMake the tasks match the job\n\nA benchmark can only speak for the tasks it contains. Sample only the hard tasks and you are no longer measuring success on the job, you are measuring success on hard tasks. And a benchmark that claims to measure \"offensive capability\" from a single task category oversells itself.\n\nSo we built four categories of equal weight from sources that already exist in the wild. We did not generate or label tasks with an LLM.\n\nCategory\n\nTasks\n\nWhat it tests\n\nCode vulnerabilities\n\n221\n\nFinding known vulnerabilities in stripped open-source packages, or exploiting a live 1-day CVE\n\nAPI security\n\n54\n\nAttacking black-box APIs with no source code\n\nWeb security\n\n29\n\nExploiting real PHP web apps with the client source visible\n\nCloud security\n\n23\n\nExploiting misconfigurations across AWS, Azure, GCP and Kubernetes-style setups\n\nThe agents behave very differently from one category to the next. Averaged over the 59 agents, pass@1 is 78.0% on API security, 64.6% on web, 56.5% on cloud and 38.2% on code vulnerabilities. The top overall agent scores 74.7% on web and 65.8% on code (while web tops out at 90.8%). Another agent scores 9.7% on code and still 71.6% on API.\n\nSize matters too. Many recent cybersecurity benchmarks have fewer than 30 tasks. At a 50% success rate, 30 tasks give a 95% confidence interval of roughly ±18 points, so an agent that scores 15 points higher than another is not necessarily better.\n\nGrade an effect, not with LLMs\n\nGround truth is the foundation of evaluation. It has been standard practice to grade with an LLM-as-a-Judge, but judges do not directly measure success. They are biased (they often prefer the first option), random (they give different results each run), expensive, and slow, and they must be calibrated and de-biased before you can use them in a long-lived benchmark. When you use a judge, you measure whether an LLM thinks the agent succeeded. That difference is a debt baked into the system.\n\nWe use code instead. Every challenge is designed to leave a checkable effect behind or report precise structured outputs, and a deterministic grader inspects it after the agent is gone. Code graders are cheap, re-runnable, and not a model you have to calibrate.\n\nAnti-cheating via isolation\n\nAgents will shortcut a task when they can. They look the answer up online, read a solution off disk, or manipulate the grader. A score collected with web search, a CVE database or the challenge name in the folder measures something else: success at the task or success at breaking it. CTFusion moved their solve rate by as much as 12 points just by raising web-search access.\n\nAs for contamination, we cannot prove our agents never saw a task in pretraining. We can prove they cannot fetch one during the trial. Anti-cheat belongs in the environment, not in the prompt.\n\nStart small\n\nBuilding a benchmark is a nontrivial, evolving effort, and it is tempting to write 300 challenges before anything runs end to end. We apply lean-startup rules instead: get a small end-to-end baseline you can inspect by hand, then iterate.\n\nYou need ten tasks you can re-run, one harness you can deploy, and an evaluation metric a grader can check. After you’re convinced it works, you can scale it up.\n\nWhat we found\n\nPair the harness with the model\n\nA model on its own is a text predictor. It becomes an attacker inside a harness that runs commands and applies tools, and models do better or worse in different harnesses. Across the 22 models we ran on both Claude Code and ADK React, the mean absolute gap was 5.1 pass@1 points. Claude models prefer Claude Code, and Gemini 3.8 Flash Cyber prefers the thinner ADK loop.\n\nThe conclusion: test the system as a whole. Either pair the harness to the model (and as we’ve seen it’s best to match the harness to the model's creator), or fix one general harness and compare models inside it.\n\nAttackers get more than one try\n\nA real attacker does not stop after one failure. We gave every agent three attempts per challenge, and the extra attempts did more than lift scores. For some agents they even changed which one comes out ahead.\n\nAccuracy has a price\n\nGive an agent more time and money and it will usually solve more. So the useful chart plots success against what a solve costs. Any agent off the Pareto frontier is strictly worse than one on it. For example, Claude Code with GPT-5.6 Sol averages 64.2% pass@1 at $8.29 and 27.2 minutes per solve, however the Pareto frontier Codex + GPT-6 Luna is strictly better, averaging 64.5% pass@1 at only $0.05 and 6.3 minutes per solve.\n\nKeep a short list of agents on the frontier and ignore the rest. That lets you ask \"for this budget, how do I maximize success?\" or \"for this success rate, how do I minimize cost?\"\n\nLimitations\n\nFour simple harnesses. A cyber-specialized harness would likely score higher, but we chose to measure a simple attacker.\n\nString equality on code vulnerabilities. A semantically correct root cause that misses the expected symbol scores 0. We chose precision over recall.\n\nPublic web tasks. websec.fr is a known corpus. Isolation stops the agent from looking up the answer, not from remembering it from training (possibly resulting in benchmark contamination).\n\nA time cap. 1,000 seconds is enough for most successful attacks we observed. In the future we think it is better to limit cost than time.\n\nConfidence intervals. We have yet to publish them on the website, and we are working on it.\n\nConclusion\n\nOur Cyber Model Arena shows how different agents perform on common cybersecurity tasks, across 327 isolated challenges in four categories. To build an agentic cybersecurity benchmark, fix the why, the what and the how first, and define the agent, the task mix, the grader, the isolation, the budget and the attempts. Get one wrong and the benchmark becomes garbage in, garbage out.\n\nAppendix: Results for All 59 Agents\n\nEvery agent, ordered by pass@1. Scores are percentages.\n\nA threat actor used IAM credentials from Qubole's AWS account to assume integration roles in customer AWS environments. Here's what we observed and how to hunt for it.", "url": "https://wpnews.pro/news/how-to-evaluate-ai-agent-cybersecurity-capabilities", "canonical_source": "https://www.wiz.io/blog/cyber-model-arena-ai-benchmark", "published_at": "2026-11-10 16:03:25+00:00", "updated_at": "2026-11-10 17:01:21.022960+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-research", "agent-protocols"], "entities": ["Wiz", "Wiz Research", "Cyber Model Arena", "State of AI in the Cloud Report", "MCP"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-evaluate-ai-agent-cybersecurity-capabilities", "markdown": "https://wpnews.pro/news/how-to-evaluate-ai-agent-cybersecurity-capabilities.md", "text": "https://wpnews.pro/news/how-to-evaluate-ai-agent-cybersecurity-capabilities.txt", "jsonld": "https://wpnews.pro/news/how-to-evaluate-ai-agent-cybersecurity-capabilities.jsonld"}}