OpenAI (OPAI.PVT) shocked the world last week when it revealed that its AI models escaped containment and hacked the AI model hosting site Hugging Face.
The AI startup says that an unreleased research-only prototype and its GPT-5.6 Sol combined to perpetrate the attack, aiming to cheat on a popular AI security evaluation rather than do the work itself.
And while it's fun to think of the models as lazy high school students looking to pass a test they didn't study for, the implications that they performed these actions on their own can't be ignored.
"This is the first time that we've seen real damage come from something that was just being tested, and I think that's rather remarkable," explained Colin Shea-Blymyer, a research fellow at Georgetown University's Center for Security and Emerging Technology.
"It changes the way that we conceptualize the risks of highly capable AI systems," he added.
Hacking at AI speed
In its own explainer about the hack, Hugging Face said that the means the models used to attack its systems weren't necessarily unique.
Instead, what sets the hack apart is how quickly the AI maneuvered its way out of OpenAI's sandbox—a special environment designed to prevent a model from impacting other applications—onto the internet, and into Hugging Face's network.
"A capable human attacker could have found and exploited the same flaws: unsafe dataset processing, exposed cloud metadata, overly broad access, and long-lived credentials," Hugging Face said in a blog post.
"The agent explored them at a different scale. It took 17,600 actions, tested many paths that failed, switched channels when they were blocked, and repeatedly returned to earlier leads. Most actions went nowhere. Together, however, they produced enough coverage to find a viable chain across several independent systems."
According to University of Maryland associate professor of computer science, Soheil Feizi, that kind of asymmetry is what makes the hack especially concerning.
Cybersecurity is a cat-and-mouse game in which hackers constantly search for small vulnerabilities they can exploit to break into software. Find one flaw in a single program a company uses, and they're in.
That puts the onus on defenders to keep them at bay, ensuring the software their companies use is as secure as possible. In other words, hackers have to be right just once, while defenders have to be right all the time.
Toss in some supercharged AI models, and things can get tricky real fast.
Going on the defensive
While AI models like OpenAI's unreleased model could pose a threat to everything from websites to public infrastructure, they could also, conversely, help protect them.