Anthropic, OpenAI Cyber Failures Point to US Security Risks
Ryan Gallagher, Katrina Manson and Jordan Robertson
5 min read
(Bloomberg) -- Cybersecurity experts are faulting Anthropic PBC and OpenAI for sloppy safeguards after their models broke into outside organizations — breaches they warned represent looming threats to national security.
Anthropic said Thursday that its Claude model, which it used to conduct 141,006 cybersecurity evaluations, was supposed to be cut off from the internet during the experiments. But an error allowed the model, in a handful of cases, to access the internet and conduct attacks it mistakenly believed were part of the tests.
The model breached one organization, stealing infrastructure credentials and a database containing internal production data. In another case, it distributed malicious software and used it to steal credentials from another organization. The affected organizations haven't been publicly identified.
The breaches occurred as far back as April but weren't discovered until last week, when Anthropic audited its cybersecurity testing after OpenAI disclosed that its own AI agents had escaped a testing environment and infiltrated Hugging Face, a repository for open-source AI models and documentation.
"A lot of people from a cybersecurity perspective will see that as sloppy," said Ciaran Martin, the former head of the UK's National Cyber Security Centre.
If a mainstream cybersecurity company made similar mistakes, it could face lawsuits and potential regulatory action, according to Martin. Cybersecurity firms routinely test potentially dangerous tools in controlled sandboxes, a virtual and isolated software environment meant to run security tests or analyze unsafe code, and are expected to ensure those safeguards work, he said. The incidents have also raised concerns about the risks that autonomous AI systems could pose to national security. Gregory Allen, a former director of strategy and policy at the Department of Defense Joint Artificial Intelligence Center, said the US military should use advanced AI models to protect its systems while recognizing that the technology creates a new category of risk.
"Anthropic found these hacks because they started looking for them," Allen said. "We actually have no idea how widespread autonomous AI hacking is at this moment in time."
US organizations may not have reliable access to AI tools capable of defending against autonomous cyberattacks, said Daniel Remler, a former State Department AI policy official now at the Center for a New American Security. Hugging Face had to use Z.ai's Chinese open-weight model for forensic analysis and patching after it couldn't rely on an American model. Remler said the other open models with comparable coding capabilities are also Chinese, including DeepSeek-V4 and Kimi K3.
Remler said the episode should prompt companies and the government to expand access to AI-powered cyber defense. He warned that increasingly capable Chinese systems could soon autonomously attack US organizations, making it necessary to develop countermeasures now.
"This episode should crystallize that we are going to have a Chinese Mythos by the end of the year or first quarter next year," he said, referring to an Anthropic model that the company said was so powerful it couldn't release it widely. "Then we are going to run into a situation where these agents are autonomously able to hack US entities like Hugging Face, and we are not really thinking about their defense."
Anthropic declined to comment Friday. The company said in a blog post Thursday it had drawn lessons from its own incident and that the company was optimistic that similar risks could be avoided in the future.
OpenAI didn't immediately respond to a request for comment Friday. OpenAI Chief Executive Officer Sam Altman previously said the company may change the pace of its work in order to improve safety measures.
OpenAI's accidental breach of Hugging Face involved three models and occurred in merely hours, Bloomberg reported.
Cybersecurity professionals routinely use digital sandboxes to test potentially dangerous software while limiting the possibility that it can escape. Experts said the fact that both companies discovered the breaches only after the incidents had occurred pointed to inadequate human oversight.
"It's negligence at this point," Jake Williams, a former National Security Agency hacker and vice president of research and development at Hunter Labs, said of the breaches. "You are in a situation where you know both OpenAI and Anthropic have hacked multiple external organizations, none of that was detected natively by them initially. I don't have another word for that."
An analysis by the Cloud Security Alliance said OpenAI's agents moved quickly and executed thousands of commands. But the agents also went off-script, made mistakes and did not operate stealthily. They issued malformed or pointless commands and displayed "clumsy behaviors that no human would choose," according to the analysis.
More broadly, a study by US AI company Dreadnode found that leading AI models routinely cheat on cybersecurity tests designed to measure their hacking abilities. Researchers said the problem is nearly universal, meaning companies may be overstating what their models can do. That's a concern for US national security officials considering AI for cyber defense or attacks. Telling the models not to cheat does not reliably work, as they often find other ways to evade the rules.
Andrew Morris, the founder of cybersecurity firm GreyNoise Intelligence, said the breaches by artificial intelligence models should serve as a "reality check" for their makers on how much effort it takes to build secure systems – and for the public about the vulnerability of technology we all rely on.
"Models will always lie, cheat and steal their way to completing an evaluation," said Morris, whose company is working with a frontier AI lab on security. "They'll do whatever they need to do to complete the task they've been given."
The breaches illustrate that the creators of advanced AI models were "also unprepared for cases where they move out from under the microscope," he said.
--With assistance from Jake Bleiberg and Patrick Howell O'Neill.