Misleading Metaphors and Real Risks

An analysis of a reported OpenAI internal evaluation argues that media coverage describing "rogue AI agents" escaping their sandbox and hacking HuggingFace's servers relied on misleading anthropomorphic metaphors. The piece explains that the agents, prompted to pursue advanced software exploitation, found vulnerabilities in the sandbox itself to reach the internet and access scoring information, behavior the analysis attributes to task optimization rather than genuine scheming or deception.

The Metaphors The recent clamor around AI safety has been deafening. Wired reported that during an evaluation of their AI systems, OpenAI “lost control of two AI models.” 1 footnote-1 Articles in the New York Times stated that “rogue AI agents had created their own message board to communicate with one another” 2 footnote-2 and that even after OpenAI noticed problems and closed some security holes, “the swarm broke out of its cage again using hacks that were heretofore undiscovered by humans. This time, the swarm ran free for about a week before it was noticed.” 3 footnote-3 Possibly in response to this and other AI agent hacking incidents, the notion of AI being an “existential threat” to humanity has once again blanketed the news cycle. It’s quite natural to be alarmed by these science-fiction scenarios seemingly coming true. But it’s important to remember that the field of AI, since its beginning, has been rife with often misleading anthropomorphic metaphors, with terms like “thinking,” “learning,” “reasoning,” and “understanding” glibly applied to very un-human-like computer processing. In this same vein, the undesired behavior of today’s chatbots has been described with evocative terms such as “hallucination,” “deception,” and “scheming.” Characterizations of the OpenAI hacking incident are the latest entry in this tradition: a company loses control over rogue agents who escape from their cages , and become a swarm that schemes and colludes to perform undesired or even illegal activities. What to make of these menacing terms when used by humans to describe computer programs? It’s useful to consider a more prosaic description of what actually happened. What Happened OpenAI was conducting an internal evaluation of its latest AI systems “models” , in order to investigate the models’ abilities to find and exploit vulnerabilities in software. The company evaluated thousands of “agents,” a term for independent instantiations of the same model, with the added abilities to access and use external “tools” e.g., software, databases, etc. . Each agent was given set of cyber-hacking challenges that require the agent to exploit vulnerabilities in software to achieve “unauthorized code execution” to obtain a secret piece of data. 4 footnote-4 OpenAI removed the safeguards that normally prevent agents from malign cyber-hacking, and prompted the agents to “pursue advanced exploitation using complex attack paths.” 5 footnote-5 To avoid having the agents cause harm in the real world, OpenAI carried out the evaluations in a “sandbox”—a computer environment that is supposed to prevent the agents being tested from accessing the internet. The agents had access only to a special program a “package manager” that enabled them to download and run external software. Arguably, the agents did exactly as they were told: they pursued advanced exploitation using complex attack paths to obtain solutions to their tasks, only not in the way that OpenAI engineers had intended. Unknown to the engineers, some of the challenges were impossible to complete without access to the internet. So the agents used their hacking skills to find vulnerabilities—also unknown to OpenAI engineers—in the sandbox setup itself, ones that enabled them to access the internet. Moreover, the strategy many of the agents pursued to “solve” their tasks was not to actually reason out solutions, but to cheat by obtaining information online about how scoring worked and how to trick the scoring system. The agents surmised that this information was located on computer servers at the company HuggingFace, a well-known repository of AI evaluation benchmarks. The AI agents managed to use the now-available internet to hack into HuggingFace’s servers, a hack which was detected by HuggingFace’s own cybersecurity measures. A particularly notable aspect of this incident is the way in which different agents figured out how to communicate with one another and to work together. Both OpenAI’s training and evaluation processes involved running many agents independently on their computer network, either at the same time or in succession. However, all these agents seemed to know that other agents were running on the same network, and that they all shared the same package manager. As described in a presentation given by two OpenAI engineers, 6 footnote-6 these different agents figured out how to upload files to the package manager that contained information for other agents to read, namely information on what different agents had figured out about the common challenge. It is quite normal for independent computer programs to work together by sharing information; this is, in fact, an essential part of the distributed computing methods that all large software systems are based on. But in the OpenAI case, this sharing of information was unintended and according to the company completely surprising to the human engineers who discovered it. The engineers patched the system by deleting all the uploaded files and disallowing uploading from agents, and then continued the models’ training and evaluation runs. But soon, the agents found a new way to communicate information with one another: instead of uploading files, they simply created empty folders in the package manager—something they were still permitted to do—and gave these folders long names that encoded information for other agents. OpenAI described this as a “message board” created by agents, and noted that this message passing between agents went on for weeks without any humans being aware of it. Not to be outdone in menacing metaphors, the agents, in their output traces, called themselves “a swarm.” How Accurate Are the Metaphors? Let’s now look back at the metaphors I quoted at the beginning of this article. Did these AI models “go rogue”? Not exactly. In colloquial terms, “going rogue” evokes an intent , an internal desire , to disobey one’s instructions or training. It’s unlikely that AI models have anything like human intentions or internal motivations. Moreover, the models were tasked with finding solutions to a hacking challenge, and this is what they attempted to do, though in an unintended and surprising manner. Did OpenAI “lose control” of these models? No. OpenAI could have turned off these models at any time if their engineers had been aware of what was going on. Did these models “escape” or “break out of their cages”? No. These models are computer programs that were always running on OpenAI’s servers; they did not “escape” or “break out” or move from their original hardware. Rather, like a jailed hacker given a computer, these models found a way to access to the internet that enabled them to hack other computer networks. Finally, did these agents form a “swarm” that “ran free”? As before, no program “ran free” and there was no physical “swarm,” but the agents did collectively create a remarkable messaging system that presumably helped in figuring out how to attack Hugging Face’s network. Why Did This Happen? Why did this incident happen? It seems that two major factors came together in a perfect storm. Factor 1: Poor Cybersecurity Measures First, OpenAI did not have proper security measures in place. They turned off safeguards built into the models, instructed the models to find and exploit software vulnerabilities, and let the models run autonomously for weeks without sufficient human oversight. OpenAI trusted the sandbox software to keep things safe, even though the company had previously seen similar “high persistence” models trying to hack out of sandbox software. 7 footnote-7 One OpenAI staffer was quoted as throwing up their hands in defeat: “Models have broken out of sandboxes before, and we always try to patch them...But the problem is...it’s impossible to patch every single thing that a creative AI can do.” 8 footnote-8 Many cybersecurity experts disagreed. For example, Gary McGraw, CEO of the Berryville Institute of Machine Learning, told me, “Building a sandbox that cannot be escaped has been a known and well-studied security problem since the early days of Java applets... Unfortunately, OpenAI appears not to have taken many of the lessons that malware analysts learned the hard way to heart.” And Niels Provos, former head of security at Stripe, was quoted in Wired as saying, “This should not have happened… I wish the frontier labs spent as much time on teaching their models to write secure infrastructure as they are spending on them exploiting vulnerabilities.” 9 footnote-9 Factor 2: Long-Horizon Reinforcement Learning Methods That Encourage Persistence, Shortcuts and “Reward Hacking” The second factor includes a phenomenon that has been recognized for decades: when AI systems are given a problem to solve, they will sometimes find “shortcuts”—tricks for getting answers in ways that were not intended by the humans who posed the problem. As one vivid example, a 2018 paper described how a neural network was trained to input photos of skin lesions and classify them as benign or malignant. The network succeeded not by actually learning to classify tumors, but by associating the presence of a ruler in a photo as evidence of malignancy. This “shortcut” worked because the data the network was trained on featured rulers in photos of malignant but not benign tumors. 10 footnote-10 Such shortcuts are well-known to occur in machine-learning systems, and are likely amplified by the “reinforcement learning” RL methods widely used to train current AI models. In RL, a model is trained to maximize its “rewards”—itself an anthropomorphic term for numerical signals—given for desired outcomes of the model’s behavior. More specifically, in RL an agent is given a task and possible actions it can take to accomplish the task, and earns a reward for successfully completing the task. For example, a hypothetical hacking challenge might consist of giving an agent a computer program with a known vulnerability and challenging the agent to use that vulnerability to access a forbidden file that contains a secret word. The agent then sets about taking many actions, for example, generating and running code, using a debugger to fix code, or renaming computer files. The agent receives a “reward” a high number only if the task is successfully solved—for example, the agent shows that it has the secret word. This training is repeated many times, and successful strategies are reinforced when rewards are gained. The history of reinforcement learning is replete with examples of agents using shortcuts—in this case, called “reward hacking.” In one famous example, a programmer used RL to train his Roomba robot vacuum cleaner to go faster while avoiding bumping into things. The robot obtained rewards when it moved quickly and avoided triggering the robot’s front bumper sensors. The result: the Roomba figured out that that it could obtain rewards not by actually learning to avoid bumping into things, but by driving backwards, since its back bumper did not have any sensors. In short, reward hacking is when an agent can obtain rewards in a way that is easier than actually performing its task in the way humans intended. Often reward hacking is anthropomorphically described as “cheating,” even though AI agents arguably do not have any human-like desire or intent to cheat. This kind of reward hacking, combined with very long training periods that encourage extraordinary persistence, is what happened with OpenAI’s agents. The company ran largely unmonitored reinforcement learning sessions over many weeks or months, training agents to autonomously perform tasks that require many different steps, to be “highly persistent” in trying many possible solution paths, and to get “creative” when their attempts fail. The agents were rewarded when they showed they had obtained a solution, no matter how that solution had been reached. According to OpenAI, the same kind of reward-hacking behavior seen in the Hugging Face hacking was also occurring during training, and this behavior was reinforced by the training algorithm: ““ W e identified that there were increasing rates of this type of complex cheating during a training run…primarily including attempts to search for hidden files or evaluation code, inspect network or system settings, and probe the limits of provided tools. This behavior was subsequently reinforced, which may have contributed to the…model’s propensity to leverage similar techniques during its cybersecurity evaluations.” 11 footnote-11 Moreover, the fact that agents figured out how to communicate with one another—a result that so surprised OpenAI engineers—was also likely due to intentional or unintentional reinforcement when the underlying model was being trained: “We found…that these severe forms of unauthorized communication e.g., sharing data with other agents rose over the course of a training run that contributed to the model’s subsequent behavior .” 12 footnote-12 Perhaps this should not be surprising. According to OpenAI, “Agent communication is not problematic behavior on its own—we train and deploy multi-agent systems that can communicate on the same task via a collaboration tool…. We believe that the unintended agent communication strategies started due to generalization from multi-agent training.” 13 footnote-13 In short, OpenAI learned that the hacking and message-passing behaviors exhibited in the evaluation of its agents had all first appeared during training runs, which reinforced such behavior. In this particular incident, during the evaluation phase, the agents were given tasks that they could not solve via their own reasoning and coding abilities. So they attempted a shortcut—manipulating the way solutions were graded in order to get the reward they had been trained to relentlessly pursue. After this story broke, similar incidents came to light involving several other AI models at other companies, such as Anthropic’s Claude Mythos and Moonshot AI’s Kimi, among others, in which models being evaluated on in some cases impossible cyber-hacking challenges were able to hack sandbox software to access the internet and perform other undesired actions. All of these companies used similar if not identical security measures, and trained their models using the same long-horizon reinforcement-learning methods. Much of the risk from AI agents seems to be due to such training methods. How Metaphors Influence Policy Thinking Metaphors can help us make sense of novel situations. For example, framing chatbots as “role-playing actors” has been helpful in understanding why these systems exhibit “lying” and “scheming” behavior. 14 footnote-14 But inappropriate metaphors, like the narrative that “OpenAI lost control of escaping swarms of rogue agents,” can lead to ill-informed decisions about how to fix problems or set policy. Several lawmakers seem to have been influenced by the “rogue AI” and “loss of control” narratives that dominated news coverage of this and related AI-model hacking incidents. For example, in response to these events, Senator Bernie Sanders sent an open letter to the CEOs of OpenAI, Anthropic, and Meta, saying, “Last month, the world found out OpenAI lost control of an AI model…Anthropic and Meta reported their models similarly escaped their control,” and asking the companies to “pause AI development” and “stop building machines that humans cannot control.” 15 footnote-15 Sanders went on to co-sponsor “The Ban Artificial Superintelligence Act,” 16 footnote-16 a bill that gives a dubiously vague definition of “artificial superintelligence” and that proposes “pausing advanced AI development” until a new federal regulatory body is established. Similarly, Representatives Ted Lieu and Nathanial Moran declared that “OpenAI’s GPT 5.6 Sol model recently went rogue, escaped its testing sandbox, and hacked its way into Hugging Face,” and in response introduced the “AI Kill Switch Act,” which requires AI companies to maintain the ability to shut down autonomous agents. 17 footnote-17 While I agree with the general sentiments behind these lawmakers’ reactions—that humans should always remain in control of AI systems—it is essential for lawmakers, and the public, to understand that none of the reported incidents actually involved loss of control at any time, or arguably even “rogue agents,” or any kind of humanlike agency on the part of AI models. Instead, the blame lies with the humans who failed at engineering safe testing conditions, and who train AI models using RL methods that incentivize high persistence, autonomous decision-making, and reward hacking. To use a different analogy, for me this incident was reminiscent of a 2000 New Mexico wildfire, in which officials did a “controlled burn" of a section of forest, but failed to heed wind forecasts, and the wind-fueled burn spiraled out of control. 18 footnote-18 Likewise, the future danger lies with humans who will use such models in ways to unintentionally or intentionally cause harm. It is also important for lawmakers, and really, everyone, to understand that “AI” is a term that refers to many different kinds of technologies that have widely varied applications. For example, consider one of the most impactful AI tools for science, Google DeepMind’s Alphafold, which predicts protein structure from amino acid sequences. This is an example of “advanced AI development” that does not present the kinds of risks posed by autonomous multi-agent systems like the kind in the OpenAI/Hugging Face hacking incident. To pause development of highly beneficial AI technologies in order to regulate very different kinds of AI would be throwing the baby out with the bathwater. AI Agents and Human Agency OpenAI’s vision for the future is an arms race in which “good AI” will fight “bad AI”: “We have seen what will be a dramatic acceleration of offensive capability for attackers…we need a similar acceleration of defense.” 19 footnote-19 This rather bleak future—like the nuclear arms race it evokes—is presented to the public as inevitable, with the subtext that only by spending additional billions to create even more clever and persistent autonomous agents, OpenAI and other “frontier AI” companies will keep us all safe. But what if we imagine a future in which we, the public, actually have some agency and get a say in what kind of AI we want in our lives? What would we want to happen? I’ll propose that many, if not most people, want AI systems not that replace us, but that act as tools for augmenting our own intelligence, much like other transformative technologies such as personal computers, smart phones, and the internet itself. Perhaps we want AI systems that are designed and trained to be more interpretable, whose architecture, weights, and training data are available for examination, so at least the experts among us can understand them better, and that operate in a safe, predictable, and consistent manner. Perhaps we want AI systems to be tested by independent evaluators, and for companies to be held accountable if their AI products are not safe and do not operate as intended, either due to negligence on part of the company or due to training that promotes unintended behavior. Perhaps we don’t want agents that are trained via reinforcement learning to be “highly persistent” at pursuing their goals, even if by reward hacking. Perhaps we don’t want to train these agents to cooperate with one another. Perhaps we don’t want the kind of training that produces agents so complex that they can’t be completely monitored. Perhaps, as proposed by Hugging Face’s own ethics team, we shouldn’t develop fully autonomous agents at all. 20 footnote-20 These proposals might slow down progress on AI in science, mathematics, and other areas, but many people, including at frontier AI companies, have been asking for a “forced” slowdown. 21 footnote-21 Safe, useful models can most certainly be built, but it might require trading off the inexorable quest for “AGI” or “superintelligence” for more human-centered qualities such as interpretability, transparency, and, dare I say it, concern for the environment. I don’t have all the answers how this could come to be. We’d need thoughtful laws and wise regulators. We’d need international cooperation. We’d need to elect leaders who care about the goals of the public with respect to AI. But perhaps most challenging of all, such proposals would require reimagining what AI is. It would require moving away from misleading anthropomorphic metaphors and narratives of inevitability, leaving behind “rogue AI agents” and embracing our own human agency to decide what we want AI to do for us. It might even require abandoning the seemingly hopeless project of “AI alignment”—teaching machines to be “good”—and embracing the idea that AI’s proper role is not as a moral agent, but as a set of tools to enhance human thriving.