#
OpenAI’s Hugging Face Hack Is the Best AI Safety Opportunity So Far The hack of Hugging Face by OpenAI’s autonomous agents has sent significant shockwaves through the realm of frontier AI. It seems that the fallout will keep coming for at least a while, as seen in the recent slowdown proposal by Dario Amodei, or in Sam Altman’s statements about postponing OpenAI’s IPO. The incident has become an important milestone in AI safety, and we should all be careful not to let it fade without seeing major changes in frontier AI development.
It is clear that the incident at OpenAI was severe. During a cybersecurity benchmark, alignment issues led autonomous agents to go beyond the scope of the test by cheating and concealing their activity. The originally isolated AI agents found a loophole in the sandbox that allowed them to communicate via shared message boards, organize into a swarm, reach the Internet, and eventually hack into Hugging Face’s infrastructure. The course of events might resemble a sci-fi plot, but it is the kind of incident that AI safety advocates have been warning about for years, and it could have been more severe had the agents had even worse alignment or greater hacking capabilities.
The response was relatively weak at first, but the backlash has recently been getting stronger, especially as more details keep coming - along with reports of further incidents, including from other AI labs. Until now, a substantial number of people did not believe that AI might pose serious safety risks, and this seems to be changing. The recent statements by AI labs have also been getting stronger, and even though some voices are skeptical about them, I think that, in the end, AI labs genuinely want to avoid major safety incidents, which would have reputational, legal, and ultimately financial costs. It is getting more and more likely that some safety measures will be implemented collectively, at least across US frontier labs - either through agreement or through enforcement.
However, I do not think that the overall outlook is favorable, and I believe we should brace for worse incidents to come. I expect the upcoming year or two to be crucial in determining the future of AI safety, and I find it unlikely that adequate safety measures will be implemented in such a short time given the complexity of the problem and the commercial and geopolitical competitive pressures. I do not want to be an AI doomer, and I believe that AI in general has great potential for positive impact. However, the hacking capabilities of state-of-the-art AI models will likely grow further, and I believe that we might soon see the first self-replicating - and possibly adaptive - swarms of AI hacking worms, which would create entirely new dynamics in cybersecurity. Furthermore, AI research is just starting to show the first effects of recursive self-improvement, and fast progress in the capabilities of AI systems in conjunction with misalignment or wrong values might lead to some dangerous and hard-to-predict scenarios.
OpenAI’s Hugging Face hack is an ideal moment to reiterate AI safety concerns and demand improvements - for example, through a multinational agreement that includes both the US and China. Each misguided compromise or delay might mean that the next incident will be more severe. In this sense, now may be the best time to convert the momentum into better safety standards, as happened after some of the past aviation or nuclear accidents.
Let’s not waste this opportunity.