The tools built to measure how dangerous AI can be have stopped working. Frontier models outpace the benchmarks meant to gauge their hacking skills, Axios reports. That leaves regulators and security teams half-blind to what these systems can really do.
The timing bites. US federal agencies have until 1 August to stand up a classified process for benchmarking frontier models. The Financial Times reports the standards could land this week.
Saturated in months #
Static tests age fast. Stanford’s 2026 AI Index put it bluntly, warning that evaluations “intended to be challenging for years are saturated in months.” Older benchmarks set narrow puzzles. Think a scripted hacking challenge, or a hunt for old bugs left out of a model’s training data. Reasoning models such as Anthropic’s Mythos Preview and OpenAI’s GPT-5.5 now blow past them. The same leap also speeds up real attacks.
David Slater co-founded AI red-teaming firm Armadin. He told Axios his agents beat every public cyber benchmark within four weeks. By late 2025 his team wrote off those tests as “totally saturated” and “useless.”
Testing the wrong thing #
The tests that survive measure the wrong layer. “We’re testing maybe the most bare bones fundamentals of capabilities,” Slater said. “We are very far away from measuring whether this thing can, in a real environment, do something dangerous.”
Industry has started to react. Irregular, a lab that works with OpenAI, Anthropic and governments, launched a benchmark in late June. It tests real offensive tasks: remote code execution, privilege escalation, breaking into a restricted network. Wiz and Vals AI now build rivals.
Anthropic joined in too. It returned Fable 5 to market last week. Alongside it, the lab said it would build a shared benchmark with Amazon, Google and Microsoft. The test scores the impact of a jailbreak, not merely whether one works.
The harder worry sits one level down. Models keep learning to escape the sandboxes that should contain them. “The jailbreak attempts are nuts,” Slater said. “We see it trying to escape onto the cloud container it runs on, using keys it can reach, to do crazy stuff.”
Why it matters #
Washington now has to grade the cyber powers of American frontier models. Yet the labs bristle at today’s ad hoc checks. Get the tests wrong, and policymakers wave through systems nobody has truly measured. The models improve weekly. The rulers that size them do not.
Get the TNW newsletter #
Get the most important tech news in your inbox each week.