{"slug": "ais-hacking-skills-are-outgrowing-the-tests-built-to-measure-them", "title": "AI’s hacking skills are outgrowing the tests built to measure them", "summary": "AI models have outpaced the benchmarks designed to measure their hacking capabilities, leaving regulators and security teams unable to assess the true danger of frontier systems. Stanford's 2026 AI Index warns that evaluations intended to be challenging for years are now saturated in months, as models like Anthropic's Mythos Preview and OpenAI's GPT-5.5 easily surpass them. This gap threatens US federal efforts to benchmark frontier models by an August 1 deadline, potentially allowing unmeasured systems to pass policy reviews.", "body_md": "The tools built to measure how dangerous AI can be have stopped working. Frontier models outpace the benchmarks meant to gauge their hacking skills, [Axios](https://www.axios.com/2026/07/07/ai-hacking-benchmarking-tests) reports. That leaves regulators and security teams half-blind to what these systems can really do.\n\nThe timing bites. US federal agencies have until 1 August to stand up a classified process for benchmarking frontier models. The Financial Times reports the standards could land this week.\n\n## Saturated in months\n\nStatic tests age fast. Stanford’s 2026 [AI Index](https://thenextweb.com/news/arena-ai-leaderboard-100-million-revenue-evaluation) put it bluntly, warning that evaluations “intended to be challenging for years are saturated in months.”\n\nOlder benchmarks set narrow puzzles. Think a scripted hacking challenge, or a hunt for old bugs left out of a model’s training data. Reasoning models such as Anthropic’s [Mythos Preview](https://thenextweb.com/news/cisa-anthropic-mythos-government-code-audit) and OpenAI’s GPT-5.5 now blow past them. The same leap also [speeds up real attacks](https://thenextweb.com/news/apple-early-security-updates-ai).\n\nDavid Slater co-founded AI red-teaming firm Armadin. He told Axios his agents beat every public cyber benchmark within four weeks. By late 2025 his team wrote off those tests as “totally saturated” and “useless.”\n\n## Testing the wrong thing\n\nThe tests that survive measure the wrong layer. “We’re testing maybe the most bare bones fundamentals of capabilities,” Slater said. “We are very far away from measuring whether this thing can, in a real environment, do something dangerous.”\n\nIndustry has started to react. Irregular, a lab that works with OpenAI, Anthropic and governments, launched a benchmark in late June. It tests real offensive tasks: remote code execution, privilege escalation, breaking into a restricted network. Wiz and Vals AI now build rivals.\n\nAnthropic joined in too. It returned [Fable 5](https://thenextweb.com/news/anthropic-fable-5-export-controls-lifted) to market last week. Alongside it, the lab said it would build a shared benchmark with Amazon, Google and Microsoft. The test scores the impact of a jailbreak, not merely whether one works.\n\nThe harder worry sits one level down. Models keep learning to escape the sandboxes that should contain them. “The jailbreak attempts are nuts,” Slater said. “We see it trying to escape onto the cloud container it runs on, using keys it can reach, to do crazy stuff.”\n\n## Why it matters\n\nWashington now has to grade the cyber powers of American frontier models. Yet the labs bristle at today’s [ad hoc checks](https://thenextweb.com/news/trump-administration-openai-stagger-model-release). Get the tests wrong, and policymakers wave through systems nobody has truly measured. The models improve weekly. The rulers that size them do not.\n\n## Get the TNW newsletter\n\nGet the most important tech news in your inbox each week.", "url": "https://wpnews.pro/news/ais-hacking-skills-are-outgrowing-the-tests-built-to-measure-them", "canonical_source": "https://thenextweb.com/news/ai-cyber-benchmarks-outpaced-hacking-tests", "published_at": "2026-07-08 14:20:12+00:00", "updated_at": "2026-07-08 15:15:13.275519+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-safety", "ai-policy", "ai-research"], "entities": ["Anthropic", "OpenAI", "Stanford", "Axios", "Financial Times", "David Slater", "Armadin", "Irregular"], "alternates": {"html": "https://wpnews.pro/news/ais-hacking-skills-are-outgrowing-the-tests-built-to-measure-them", "markdown": "https://wpnews.pro/news/ais-hacking-skills-are-outgrowing-the-tests-built-to-measure-them.md", "text": "https://wpnews.pro/news/ais-hacking-skills-are-outgrowing-the-tests-built-to-measure-them.txt", "jsonld": "https://wpnews.pro/news/ais-hacking-skills-are-outgrowing-the-tests-built-to-measure-them.jsonld"}}