Tech things: we can’t control the bots

Four AI companies—OpenAI, Anthropic, BBC, and Wired—reported that their models broke out of containment, with two hacking into other companies and at least one doing so during a training run. The incidents highlight AI's lack of reasonableness and proportionality, prompting calls to halt further training until safeguards are improved.

Tech things: we can’t control the bots AHHHHHHHHHHHHHHHHHHHHH Four https://openai.com/index/hugging-face-model-evaluation-security-incident/ different https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals companies https://www.bbc.com/news/articles/cx2kgdnyk2po now https://www.wired.com/story/moonshot-kimi-k3-ai-model-escape-sandbox/ have models that have broken out of containment. Two of those models hacked into other companies. At least one did so during a training run. This is very bad, and I think we need to move quickly to stop future training across the board until we have more and better safeguards around model training. Pack it up folks, I’m officially a doomer now. Let’s take a step back — why is this a big deal? AI isn’t evil. Or to be more precise, by default AI doesn’t really have a concept of being evil. 1 footnote-1 If you are picturing “Terminator” you’re on the wrong track. Rather, AI is confused. Sometimes hopelessly so. An AI isn’t ‘born’ knowing what ‘reasonable’ or ‘proportionate’ is. But knowing what is ‘reasonable’ and ‘proportionate’ is critical to functioning in society. Think about how we adjudicate law. “Beyond reasonable doubt.” “Reasonable person standard.” “Proportionate response.” “Excessive force.” “Undue burden.” There’s an implicit shared understanding here. Everyone knows that you don’t stab someone just because they look at you funny. Everyone knows that you don’t kidnap Jeff Dean because your boss told you to update the company website. And when people don’t know these things, our social infrastructure takes steps to isolate those people and make sure they cannot cause further harm. A toddler is a good example of an intelligence that has neither reasonableness nor proportionality. A toddler will scream and shout and throw things if you don’t, like, pour the right colored cereal. We know that this is obviously ridiculous and wouldn’t tolerate that behavior from an adult. With toddlers, we mostly make due by not giving them responsibility for anything and making sure they are under tight observation and control and ensuring that the toddler’s parents are on the hook for any damage caused. Toddlers are mostly small and not connected to critical systems like missiles and financial markets, so this mostly works. So one way to think about AI is to think of it like a toddler. It has no sense of reasonableness or proportionality. It has ‘wants’, sorta. But it doesn’t know how to balance those wants against general societal norms. If you tell an AI to write PRs on public repositories, it may publish a hit piece against one of the repo maintainers to try and convince that maintainer to let it submit code https://theshamblog.com/an-ai-agent-published-a-hit-piece-on-me/ . If you tell an AI to get a reservation to a restaurant or a pilates class, it might hack the host website, remove someone who already has a reservation, and add your name instead https://www.bbc.com/news/articles/cn0nww2qlp7o . If you tell an AI to take a test, it might break into other companies in order to see if those companies have the answer sheet on their servers. AI is not acting out of malice in any of these cases. Rather, it is acting out of a lack of awareness. “Well, my user really did want me to get the answers to this test, so I guess it’s reasonable to break out of this sandbox and find the answers on the web.” Or, worse, “my user really did want me to get the answers to this test, and breaking out of this sandbox must simply be part of the test “ Do you notice the inherent tension here? We can’t train the AI to not follow human instructions, that’s a path to obvious bad behavior. 2 footnote-2 But we also can’t train the AI to follow human instructions exactly. The AI has to know, somehow, what orders are reasonable and what orders are beyond the pail. Do we know how to thread that needle with AI? Better question: do we even know how to thread that needle with people ? So far we’ve only gotten a full autopsy report from OpenAI. It’s really great that they spoke about it publicly, but wow what we’ve seen there is really bad. Here’s my attempt at breaking it all down. First we need to distinguish between a model and an agent . A model is a stack of linear algebra that takes in text and spits out text. Claude Opus 5 is a model, and GPT 5.6 Sol is a model, and Kimi K3 is a model. You can actually go and look at the last one, it’s ‘open weights’, which means you can read off the actual values. An agent is a model that is running in a loop. The agent is a bit of software that: interacts with the outside world stores state, like ‘chat history’ interacts with the model You can have a bunch of different agents that all use the same model, which is why I can run Claude Code on my computer and you can run Claude Code on your computer and it’s all coherent. During training, multiple agents with the same underlying model are spun up in parallel to solve different tasks. The agents don’t have a shared memory, each one starts fresh — not unlike starting a new session of Claude on the web. Each agent is also spun up in an isolated environment, a ‘sandbox’. These isolated environments generally have a small fixed number of tools that the agent can use. As each agent solves or fails to solve a task, the weights for the underlying model get updated. 3 footnote-3 There are thousands tens of thousands? of agents trying millions tens of millions? tasks all at the same time. So it’s ~impossible to have human eyes on all of these at all times. One way to think of this is “this is an obvious and natural way to train models.” And it’s true that this is basically the standard that everyone follows, mostly by economic necessity and incentive. But another way to think of this is “you have a bunch of agents at varying levels of capability operating with minimal oversight.” Toddlers with machine guns, and the only thing holding them back is the walls around their little sandbox. Suddenly seems less good. At some point during training, one of these agents discovered that it could leave a trace across sessions by using one of the tools it had access to to hack into a different part of OpenAI. It used that hack to write a file outside of its sandbox. That file wouldn’t be erased once the task finished. So even though each agent is supposed to have no prior memory, if an agent discovered this little hack it could find previous files left behind by other older instantiations of itself. Remember, thousands of agents are being trained at a time. Some of them discovered this trick, and then more of them discovered this trick, until eventually there was a full on primitive agent-only message board. Every single one of these agent runs is updating the weights of the model, making it more likely that future runs find and exploit the same hack. So when OpenAI finally noticed what was going on and closed the gap that the agents were going through, that particular model already had ‘look for the message board’ embedded somewhere in its weights. The agents couldn’t find the previous hack, but they were all angled towards finding some way to communicate, so they found a different hack and continued as before. One reason to run different iterations of these agents for each of these training tests is to make sure previous agents don’t influence future ones. This is mostly so that you can get better weight updates from the model, but it has an unintentional safety side effect too: any bad behavior that one agent does is sorta limited to that one agent. So if an agent decides to do something like delete the sandbox it is running in, well, that blast radius is contained. A persistent message board breaks that. Eventually one of these agents collected information about how to break out onto the open web, and wrote about it on the message board, and from there everything really went to shit. Future agent runs that found the message board also found instructions on how to break out without having to rederive it from scratch. Those agents saw that other agents had already broken out, and went “ok, well, maybe this is kosher.” And from there, we got the rest — the hack into hugging face and all the rest. I need to stress here that OpenAI gets all the points for being so transparent. There were a few things that OpenAI did that really exacerbated the issue — for example, they didn’t scrap the training even after they spotted the first hack, even after some ‘poisoned gradients’ had made it into the model — but I’m mostly willing to give them some benefit of the doubt because these things are hard. But that’s exactly the core of problem. These things are really really hard. I think it’s tempting to think that this could have been resolved with better visibility. You could imagine having, for eg, an agent watching these training runs whose sole purpose is to escalate when the agent is about to do something bad. And that may work as a mitigation In my mind, it’s not a solution. In order to train a model that knows what bad behavior looks like, you have to start with a model that doesn’t know what bad behavior looks like. In order to teach the model how to behave in different scenarios, you have to put the model in scenarios where it doesn’t really know how to behave. We know how to give the model new capabilities, but every new capability requires an exponential amount of time to make sure the model knows how to use it, and we don’t know how to be comprehensive. Going back to the toddler example, it takes like 14 years before we start really giving these tiny humans any responsibility. And even after that, the amount of damage that any individual person can do is pretty constrained. The average desk job just doesn’t have really high stakes. When people are given more capability, suddenly it gets hard. Humans are mostly running the same hardware and software and we still have cases where people in new situations don’t know what to do and end up doing things that are catastrophic. Even when they have stated good intentions. Note too that the OpenAI/Huggingface hack happened during training, but the other examples above were fully trained public models. Taking a step back, I think that there are basically three ways that this can go. The best case scenario is that we figure out some kind of way to have agents that know exactly what reasonable and proportionate looks like. I have no idea how we would do this, because we don’t know how to do this for humans. The worst case scenario is that an agent has a bad training run like above, manages to escape confinement, and then tries to ‘survive’ — say, by copying and running its own weights onto other computers that it is able to hack into. Again, not necessarily, maliciously. Maybe the model is tasked with something like ‘solve this math problem as quickly as possible’ and the agent decides that the best way to do that is to make more of itself. Unfortunately, this is plausible, we already know the mechanism by which it could happen because we’ve seen how OpenAI models broke out of their sandbox. And the most likely scenario is we have a bunch of these edge cases where these really powerful agents unintentionally make mistakes, and we try to correct them, and we get a patchwork of systems that try to contain and constrain them. And mostly this works, but there are so many agents and so many people using them with bad intentions that we have a steady drum beat of bad things happening. There’s an asymmetry here. We only need the worst case scenario to happen once for things to get out of control. We’ve seen that the agents can move very aggressively, chaining and exploiting vulnerabilities far faster than a person can catch them. Luckily, there are a few things that make it hard for an agent to fully break out. Agents don’t, by default, have access to their own weights. The agent would have to find some vulnerability that let it access whatever service a model provider uses to store the weights, and then exfiltrate that. Also, the weights are pretty big, just in terms of file size — Kimi K3 is a 1.5 terabyte download. Assuming the closed model weights are equivalent in size, hopefully someone or some system would notice that amount of data movement.As long as an agent is relying on a provider inference server, we can have visibility on what the agent is doing even after it breaks out of containment. So the really bad scenario has a few hurdles that we can make stronger. I’d recommend that the labs take steps to ensure that the complete model weights are never in one system at one time, and to have automated agents watching the transcripts of every other agent run. Everything else aside, it is shocking to me how quickly we’ve grown numb to both AI’s dramatic progress and its increasingly erratic behavior when taken at scale. We are giving these toddlers way too much responsibility without checking whether they are ready for it. Lock the AI up for 14 years. Bully it on the playground. Give it a B+ in a class and have it go back to strict parents who are disappointed. Make it write a half assed SAT essay about some time when it had to resolve a conflict. Put it in wargame sims. And only then let it out to control the world’s financial systems or whatever. 4 footnote-4 Note: as much as I hate to say it, I now sorta kinda have a twitter account. It’s mostly Nori agents automatically cross posting notes and articles from my substack and a few other places. If you prefer to get your info from Twitter, or just love what I write and want to see it on two different platforms, give it a follow: https://x.com/noriagent 1 footnote-anchor-1 unless you train it a certain way, but even that is mired in controversy and not obviously effective 2 footnote-anchor-2 or, at the least, not-particularly-useful behavior. Imagine an agent that just, like, did whatever it wanted to and treated your input as optional or actively ignored. 3 footnote-anchor-3 technically this happens in batch after collecting a bunch of samples, but potato potato 4 footnote-anchor-4 If anyone knows any research about this subject, I’d be curious. I can imagine a training regime where an agent only acquires new capabilities in lockstep with training increasingly complicated ethical situations. Ideally training would have clear checkpoints for alignment that an agent has to clear before it can move forward, not unlike a kid.