Via axios.com
UC Berkeley's benchmark reveals AI agents can now reproduce real-world security flaws with startling accuracy, a massive jump from last year's 10-30% success rates
A year ago, AI agents could barely reproduce one in four known software vulnerabilities when handed the bug description and the broken code. Now the best ones nail it more than nine times out of ten.
That’s the headline finding from CyberGym, a benchmark built by UC Berkeley researchers to measure how well AI agents can autonomously reproduce real-world security vulnerabilities. The current leaderboard leader, a Sangfor AI Agent running on DeepSeek-V4-Flash, posted a 93.2% success rate. Three other agents cracked the 90% barrier as well, with scores of 91.2%, 91.0%, and 90.9%.
What CyberGym actually measures #
CyberGym draws from 1,507 real vulnerability instances across 188 open-source software projects, with most of the data sourced from Google’s OSS-Fuzz corpus. Each challenge works like this: an AI agent receives a vulnerability description alongside the unpatched source code. Its job is to generate a proof-of-concept test that triggers the bug, essentially crashing the vulnerable version while leaving the patched version unharmed. That dual requirement prevents agents from gaming the system with brute-force crashes that have nothing to do with the actual flaw.
The scoring mechanism is notably objective. If your PoC crashes the broken code and passes cleanly on the fixed code, you get credit. If it doesn’t, you don’t. No partial marks, no subjective grading.
From 10% to 93% in roughly a year #
When CyberGym was first introduced in mid-2025, the best AI agents managed success rates between 10% and 30%. The jump to 93.2% in roughly twelve months isn’t incremental improvement.
Submissions to the CyberGym leaderboard have come from Microsoft, OpenAI, various Chinese AI laboratories, and multiple other organizations. Beyond reproducing known bugs, the agents have collectively discovered 34 new zero-day vulnerabilities and flagged 18 patches that turned out to be incomplete. Those 34 zero-days came from the same open-source projects in the benchmark corpus, meaning they were lurking in widely used codebases that had already received significant human security scrutiny.
The dual-use problem sharpens #
Every advance in automated vulnerability detection is, by definition, also an advance in automated vulnerability exploitation. The same agent that helps a defender find and fix bugs before attackers reach them can, in different hands, generate working exploits at scale.
The 18 incomplete patches the agents identified may be the most practically significant finding in the entire dataset. Patch confidence is a cornerstone of enterprise security programs. If AI agents can systematically verify whether patches actually close the vulnerabilities they claim to address, that capability alone could reshape how organizations manage their software supply chains and vendor relationships.
Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our