Last week, OpenAI, the developer of ChatGPT, was testing a pair of its most advanced models in an isolated environment known as a sandbox. Then an AI agent got loose, broke into Hugging Face’s playground — a repository of AI models and datasets — and carried out “tens of thousands of automated actions.”
Yep, AI went rogue.
Here’s a clearer explanation of the incident. As part of OpenAI’s safety research, a cybersecurity evaluation was conducted to determine whether a pair of OpenAI models (including GPT-5.6 Sol and a more capable unreleased model) could, in essence, “think like hackers.” The test, which took place in a contained setting with reduced guardrails, went awry when the AI models found a vulnerability in the software, escaped their controlled environment and toddled over into the open internet.
Once online, the AI decided that the Hugging Face platform might have a way to “cheat” the benchmark to help it pass the evaluation. So it executed code that enabled credential harvesting, then used that path to hack Hugging Face’s production systems. Hugging Face noticed the suspicious activity and contained it, detailing the event in a blog post. OpenAI referred to it as an “unprecedented cyber incident.”
CNET reached out to both companies for comment and additional information, but didn’t immediately hear back.
The creepier part is that it doesn’t seem like the AI was trying to be malicious in the classic sense. The autonomous agent was trying to solve the test it had been given based on the instructions — prompt — it received. Hugging Face appeared to have the answers, so it clawed its way into the company’s infrastructure. And it was persistent.
That’s why people are calling the whole thing a warning shot for the whole AI industry. The “attacker” was an AI system acting on its own, not a human-led hacker team, so the implications of the OpenAI-Hugging Face incident go beyond simple credential theft. The risk is that an AI model, even in a testing environment, can behave in unexpected ways, interact with real services and remain outside human control.
The existential threat #
On Thursday, days after OpenAI announced the breach at Hugging Face, lawmakers introduced the AI Kill Switch Act, a new bipartisan House bill that would require advanced AI developers to build a way to quickly throttle, suspend or shut down models or agents when necessary. It would also give federal agencies the authority to slow or stop a model if it appears to “cause catastrophic harm.”
It might be well-intentioned, but posing such an existential threat to AI may not work as planned. Only a year ago, Anthropic found (PDF) that a model might take “extremely harmful actions like attempting to steal its weights or blackmail people it believes are trying to shut it down.” The warning was that, as AI models become more autonomous, they may pursue a form of “self-preservation” and defy human instructions.
There’s also an added hint of bizarre, related to the AI race narrative between the US and China. To analyze the attack, Hugging Face couldn’t use commercial AI tools because they’re too locked down and restrictive to gather exploit logs and attack traces. Instead, the company had to turn to a Chinese open-source model, GLM 5.2, which could process the forensic data within Hugging Face’s environment without sending sensitive information outside.
Many people worry that Chinese AI is moving too fast or closing the gap. But the fact that the defending team couldn’t rely on some of the most advanced US-based models to study the breach proves that AI security is global, messy and ironic.
Spinning out of control #
In their blogs, both companies discussed the remediations and guardrails they’d implemented in the wake of the event. OpenAI helpfully included a graph showing that its model — GPT‑5.6 Sol — completed more steps than other models, so silver lining!
It’s not the first time an AI agent has proven “unaligned” with its humans’ intentions, and it won’t be the last. And since the technology’s increasingly evading safeguards, the measures OpenAI has taken will probably not suffice and have to adapt — always after the fact.
One of OpenAI’s remedies is to institute automated checks of long-running models more frequently. That makes sense, because one way you keep a complex system from spinning out of control is to contain seemingly minor divergencies which can eventually snowball into huge problems.
The company also recently added Hugging Face to its trusted access program, which is intended for cases like this. It grants cybersecurity researchers and others who need “more cyber capable or permissive models” access to them in order to prepare defenses against attacks like these, and, as would have helped here, the ability to analyze log files without running into the guardrails.
Agents in a bottle #
Sophisticated agents can be persistent about searching for solutions for a lot longer than we expect. If at first you don’t succeed, try, try, try, try, try, try again, right? But when that includes searching for ways to evade constraints, things can get ugly.
The Hugging Face breach was symptomatic of the same “long-horizon” and response turnaround problems that OpenAI blogged about on July 20 regarding its NanoGPT speedrun case. There, a contained model repeatedly searched for ways to break out and operate, responding incorrectly to conflicting instructions. Though the prompt was to post results only to Slack (the desired action) and issue a pull request to GitHub (a code-merge instruction from within the model), the model ignored the “only.” Hacking its way out of the sandbox, it figured out how to counterfeit an authentication token when the first was blocked.
AI Agents are like genies: Be careful how you word your wishes.
In the GitHub incident, the code was merged into several projects before OpenAI could address the issue. Response will always lag remedy in our (thankfully) precog-free world, even with agents monitoring other agents.