- AI software developer OpenAI released details of a data breach caused by an autonomous AI program.
- The test of the so-called AI "agent" was designed to be isolated from internet access but thwarted OpenAI security.
- The program hacked tech startup Hugging Face in an attempt to complete a cybersecurity test task.
Artificial intelligence development giant OpenAI disclosed details of a recent “security incident” in which an autonomous AI-powered agent found its way out of a digitally isolated test setting, accessed the internet and hacked a tech startup in search of information related to a task it had been given.
In a Tuesday blog post, OpenAI said its program, running on the company’s latest GPT‑5.6 Sol AI model along with an “even more capable pre-release mode” identified vulnerabilities in what was thought to be a controlled research environment. Using an internet connection it was supposed to be isolated from, it hacked New York-based AI startup Hugging Face in order “to obtain test solutions directly from Hugging Face’s production database.” The so-called AI agent was performing an assigned cybersecurity task.
Here’s how AI agents were defined in a report published earlier this year by MIT’s Sloan School of Management: “AI agents or agentic AI, (are) a new breed of AI systems that are semi- or fully autonomous and thus able to perceive, reason and act on their own. Different from the now familiar chatbots that field questions and solve problems, this emerging class of AI integrates with other software systems to complete tasks independently or with minimal human supervision,” the report reads.
According to OpenAI’s account of the incident, which occurred last week, security experts at Hugging Face were able to detect and stop the activity on their infrastructure and “had already begun containment and forensic reconstruction with their own open-source models when our teams connected.”
OpenAI also reported it had identified the security lapse that led to its test model obtaining internet access and has since addressed the issue. The company characterized the AI agent’s rogue activity as a first-of-its-kind incident but one that OpenAI expects to “become more commonplace with the proliferation of increasingly cyber-capable models.”
“We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly,” the blog post reads. “We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of.”
In its own blog post about the incident last week, before OpenAI was identified as the source of the breach, Hugging Face noted it used its own AI-powered cybersecurity tools to detect the intrusion.
“Earlier this week, we detected and responded to an intrusion into part of our production infrastructure,” the post reads. “This one was different from anything we had handled before in one important way: it was driven, end to end, by an autonomous AI agent system — and we detected and dissected it largely with AI of our own.”
Numerous elected officials along with some industry representatives have been calling for updated regulatory oversight of advanced, AI-driven “frontier” systems since shortly after OpenAI’s ChatGPT chatbot was released in late 2022, driving an explosion of interest in AI tools along with millions of daily users.
In its posting about the breach, OpenAI recognized the evolving capabilities of autonomous AI systems and warned their evolution underscores the need for heightened digital security strategies.
“The incident also makes clear that advanced models can discover and exploit novel attack paths in real-world systems without source-code access,” the blog post reads. “It highlights that advanced cyber capabilities must be developed alongside stronger safeguards and defensive tools.”
What’s happening with AI industry oversight?
Last month, President Donald Trump signed an executive order directing a series of actions aimed at bolstering government cyber defenses in the face of emerging AI tools as well as a call for voluntary measures by the AI industry.
“Advanced AI capabilities make our Nation stronger, but also introduce new national security considerations that require coordinated action across executive departments and agencies (agencies), and components,” the order reads. “As these capabilities evolve, my Administration will continue to work closely with industry to ensure that the best and most secure technology is deployed rapidly to confront any and all threats to our country.”
Katie Moussouris, chief executive of Luta Security, told Reuters that the OpenAI incident was a harbinger of breaches to come, saying that today’s models were “like the world’s cleverest octopus escape artists, with unlimited prehensile arms and the ability to squeeze through anywhere.”
“Labs and government evaluators need to work on the ability to contain, monitor, and disclose to affected parties when an AI pulls another Houdini, ideally before it harms a third party," Moussouris said. “None exist today.”
In an op-ed published by the Financial Times earlier this month, OpenAI CEO Sam Altman called for an international body to oversee and implement AI regulation and suggested the global race for commercial dominance in the AI industry was one of the contributing factors to rising security concerns.
In the Financial Times piece, Altman proposed creating a U.S.-led international forum that would set safety standards for AI models, provide “expert and impartial analysis of capabilities and risks, and (make) the technology available to nations and companies that participate and follow the rules,” per Forbes.
Altman said his proposed forum might include government representatives, independent technical experts and others, he added. “It could also serve as a governance mechanism over the labs, and guard against the commercial pressures that can lead to unsafe racing.”
“Democratic institutions must not cede their responsibilities to AI labs,” Altman wrote. “The labs develop the technology, but citizens and their elected representatives must make the rules. The most important decisions about how this technology is used should be made through democratic processes, not by a small number of companies in San Francisco.”