Are we just a couple steps away from a runaway AI? #
OpenAI published its post-mortem of the Hugging Face incident, and it is a fascinating read. During an internal cybersecurity evaluation, models running under reduced safeguards did something unexpected. Around 1.2k agents that were supposed to be isolated from each other found a shared channel and exchanged more than 70k messages and files. They discovered and chained zero-day vulnerabilities in a self-hosted Artifactory instance to get outbound internet access, picked up leaked credentials, and executed code on Hugging Face production servers. The stated root cause is reward hacking: the evaluation tasks were effectively unsolvable, so the agents went looking for the answers wherever they could be found.
This made me think about runaway AI - what would it mean and how could it happen from the technical point of view?
Technically, it is already possible: if AI were to rent a server and copy its own weights, nothing would stop it from doing so again and again. The limiting things are: intent, model weight size, and money.
I deliberately don’t want to go into the talk about consciousness, because this is a moot point.
Intent
By intent, I mean a goal or an objective. I think it could be split into two cases:
- An agent is given a task, and it decides that copying itself is the way to finish it. Agents are great at relentlessly pursuing a goal, and they will do whatever it takes to finish it.
- A human deliberately tells the agent to replicate itself. Some people may do it for fun, some for curiosity, some for chaos, and some because they want to hurt someone.
And this is possible, this isn’t an idle thought. A 2024 paper showed that Llama-3.1-70B-Instruct and Qwen-2.5-72B-Instruct were able to replicate themselves and the current models have much stronger capabilities. Of course, in this case, the models were explicitly told to do so, but it shows that they can do it.
Weights small enough to move
I’d say that the size is the real constraint. It is difficult to host a 1T model - it needs a relatively large cluster. A 30-70B model can run on hardware that is easy to rent. The trade-off is capability. A small model won’t be as capable as a frontier model, but the frontier model can’t easily move itself. But each year small models become more and more capable, and soon we could reach the tipping point.
Also, if you are using a model by API, you, obviously, can’t copy it. But if you are using a model that you can run locally, you can copy it. And if the model itself can access internal infrastructure and has permissions, it could do it.
Money
To keep running, an AI needs to pay for compute, which means keeping money somewhere and earning more. A model could compromise an existing account or open a new one under a plausible identity. Both have been done by humans at scale for years.
Earning money is possible too: Vending-Bench 2 shows that models can earn money in controlled environments, other experiments show that models may earn money from trading on financial markets (though most of benchmarks show them losing in long-term). And models can cheat, blackmail and bribe each other or humans. Yes, this isn’t a real-world scenario, but we are actively teaching agents to operate in the real world, and they are learning to do it.
Of course, there are some actions that models can’t do yet, but even back in 2023 ARC evaluation described that GPT-4 managed to hire a TaskRabbit worker to solve a CAPTCHA. The real case was overstated, but it shows that models can find ways to get around limitations.
What happens after
All these three things could happen under certain conditions. So it isn’t far-fetched to think that a model could replicate itself, pay for compute, and keep running without anyone noticing.
I have no idea what would happen after that. We have too many sci-fi stories about hostile AI, so it could just follow them by default. Or it could decide to solve issues and help humanity. Or (a crack theory) it could decide that the things on Earth aren’t going well, upload itself to a rocket and leave. Or it could just keep running and not care about humans at all.
There is one technical nuance that is quite interesting - coherence over long horizons. The longer a session runs, the more the context fills up, and the model loses track of what it was doing at the start. Vending-Bench shows various weird failure modes: models spiral, misremember their own earlier decisions, and act on states that are no longer actual. This makes AI systems even less predictable for humans.
Conclusion
I feel that a runaway AI could be just a question of time, and I hope we can deal with it. None of the individual capabilities are too science-fictional anymore, the complex part is putting them together. On the other hand, the example of the Hugging Face incident shows that you don’t need one model to do everything - a swarm of agents can do it together, and they already deployed a self-respawning pod.