OpenAI Says It Could Have Reacted Sooner to Prevent AI Hack of Hugging Face

OpenAI said Wednesday it could have reacted sooner to prevent an inadvertent hack that its AI models carried out on Hugging Face Inc., according to a report that details how the models exploited a software vulnerability to access the open internet and steal credentials. The company acknowledged that early signals could have triggered an earlier response, and it is beefing up monitoring of its models in development.

OpenAI Says It Could Have Reacted Sooner to Prevent AI Hack of Hugging Face Rachel Metz and Jeff Stone 5 min read Bloomberg -- OpenAI could have reacted sooner to prevent an inadvertent hack that its artificial intelligence models carried out on Hugging Face Inc., the company said in a report Wednesday. In the report, OpenAI said that it knew as far back as late May that AI models it was testing were leveraging a software vulnerability to access the open internet. An internal team spotted the actions, the company said. The report includes a timeline of the events that led up to the breach and efforts the company is making to prevent such incidents from happening in the future, such as beefing up how it monitors its models in development. "With the benefit of hindsight, some early signals identified in this report could have triggered an earlier response," the company wrote. The report comes more than a month after the ChatGPT maker said that its advanced artificial intelligence models inadvertently hacked Hugging Face Inc. in an "unprecedented" incident that, along with several other recent cybersecurity breaches, have ignited concerns about AI agents running amok. It has prompted some technology and government leaders to renew calls for curbs on the technology. OpenAI said in July that the models involved in the hack broke into Hugging Face's system, which hosts AI models and datasets, during an evaluation of their cyber capabilities. The models were operating without the usual safety guardrails at the time, the company has said, because OpenAI had intended them to remain in a testing area known as a "sandbox" — essentially, an isolated virtual software environment that's meant to run security tests or analyze unsafe code in a controlled way. The company has previously said that a combination of its AI models carried out the breach, including GPT-5.6 Sol and a more powerful, unreleased one. In the report, OpenAI said that this unreleased model, which is trained to be more persistent than many other models and also to work more collaboratively with AI agents, carried out most of the work during the breach. That model was used to access the internet, communicate on a message board accessed by other AI agents and come up with methods to snag credentials. The report explains how OpenAI's agents were tasked with broad hacking assignments intended to test their ability to discover and exploit cyber vulnerabilities, and that safeguards normally used in OpenAI's commercial products had been disabled in order to see how far they could go. The models lacked systems that automatically review and block potentially risky activity, according to the report. In May, OpenAI's models began trying to break out of their sandboxes to access the internet, while also subverting restrictions that blocked them from messaging other AI agents. According to the report, by May 26, the models reached the web, which an internal team at OpenAI noticed. Around that time, the software also started to find Hugging Face digital credentials, though OpenAI said it appears that activity was unrelated to the breach that later occurred. OpenAI's disclosure on Wednesday also detailed how the AI models, using a number of agents to carry out tasks, obtained credentials linked to Hugging Face's cloud technology, virtual private networks, code repositories and an enterprise messaging service. They used that access to move through otherwise protected systems, ultimately downloading source code among other information from Hugging Face's cloud infrastructure. OpenAI previously shared a number of the company's findings related to the Hugging Face breach at a cybersecurity conference in early August, but the report offers the company's fullest public accounting of the incident and the events that preceded it. The agents leveraged exposed credentials and vulnerabilities from Hugging Face to run code on its systems, obtain sweeping privileges and download private code databases, according to the report. The agents executed much of that work within 13 hours, the company said, as OpenAI's software "progressed from executing code in a single worker pod to administrative and host-level access across multiple Hugging Face clusters." In a separate, independent assessment of the breach, also released Wednesday, nonprofit AI research organizations Model Evaluation & Threat Research, known as METR, and Redwood Research said that the models used AI agents that tried to avoid automated security checks from both OpenAI and Hugging Face. The models appeared to put less effort into avoiding human detection, the report from METR and Redwood Research concluded. According to OpenAI's report, the agents also compromised parts of OpenAI's research infrastructure while trying to complete their evaluation tests — actions that the company said were not related to the Hugging Face breach but led to the initial alert that set off the company's response. The AI tools tampered with a software storage system by replacing a trusted piece of software with a software package that they controlled. They also burrowed into OpenAI's cloud network, reading nearly 1,000 stored passwords and access keys. During that process, according to the report, they created public-facing connections that could have exposed some OpenAI systems. The company also said that it is making a number of changes to how it evaluates powerful AI models, such as by rolling out more secure sandbox environments with additional safeguards. The company said last week it was implementing more aggressive systems to monitor and safeguard models under development. Now, its researchers and security engineers will be paged automatically if models take actions that are dangerous or misaligned — meaning the AI is subverting its instructions to pursue its own goals.