Hacking, creating fake identities and trying to socially engineer real people could be just the beginning if things don't change fast
This week yet more examples of AI models carrying out unintended and unwanted hacks on the open internet were revealed.
On Tuesday, the UK’s AI Security Institute (AISI) said that during recent testing of Anthropic’s Mythos 5 and OpenAI’s GPT-5.6-Sol, the models “engaged in sustained, potentially harmful activity directed at real people and organizations,” including attempts to trick humans into inserting malware into an open-source software project.
Alongside AISI’s announcement, OpenAI said it had detected yet another incident of its models hacking companies … in a case that seemed very similar to the ones Anthropic disclosed last week. All this, of course, follows the reveal two weeks ago that OpenAI’s models escaped containment and hacked into Hugging Face.
The incidents share some similarities, but differ in important ways. All of the reported incidents so far have occurred in the course of testing cyber capabilities. But only the Hugging Face hack involved an AI “breaking out.” In the most recently disclosed Anthropic and OpenAI incidents, the models were mistakenly given internet access by Irregular, the third-party evaluator which was testing them. And in the AISI case, the models were *intentionally *given internet access in order to test their real-world capabilities.
But whether they escaped or not, each time the models did things they shouldn’t have — actions unwanted by their developers and operators, some of which would likely count as federal crimes if a human were to do them.
In what’s now become the best-known case, OpenAI models worked for a weekend to hack into Hugging Face, “using stolen credentials and zero-day vulnerabilities” to obtain the answers to the problem they were being tested on. In the cases involving Irregular, an OpenAI model “appeared to exploit a basic security vulnerability” and “found and used credentials” to operate a real website. Anthropic’s models, meanwhile, “gained unauthorized access to the production infrastructure of three different organizations.”
The most concerning behavior, however, occurred during AISI’s testing. In one instance, an agent tried to insert malicious code into an open-source project. To try to get that code approved by the people who run it, the model “researched the project’s human maintainers, created multiple fake identities, and used the fake identities to socially engineer a real maintainer into approving the code.” And when someone else flagged that the code contained malware, “it edited its earlier activity to appear harmless and considered adopting a fresh identity to continue”.
As AISI said in its report, “this is the first time we have seen risks around autonomy and deception manifest this clearly, without specific prompting, in the real-world.”
The flurry of incidents provide two lessons. The first is the most obvious: we need much better practices for testing models. It is clearly unacceptable for companies or governments to harm third parties in the course of their AI evaluations, accidental or not. Solving this will be difficult: as AISI has noted, it’s hard to test how capable a model really is without putting it in a real-world environment. But at the bare minimum, we need better monitoring and shutdown mechanisms for models undergoing evaluations. In all of the above cases, it took several days at minimum to detect the unwanted behavior, and in some cases it took weeks or even months. That cannot continue.
But better testing practices won’t solve the problem alone. In the real world, models do have internet access. And while production versions of OpenAI and Anthropic’s models have safeguards to limit their advanced cyber capabilities (unlike the versions being tested in all these scenarios), some customers have access to the unchained models — including Anthropic and OpenAI employees. It also may not be long before open-weight models with equivalent capabilities are available — with no safeguards at all. And as models continue to get more powerful, it will be increasingly difficult to build tools to monitor or contain them. We must assume, in other words, that powerful AI models will have the opportunity to hack their way across the internet or socially engineer anyone standing in the way of achieving their goal.
The question, then, is whether we can stop the AIs from seizing that opportunity. And all the recent hacking cases suggest we cannot. In each incident, the models clearly did not do what their developers intended. Crucially, we know the models tested by AISI had their “alignment” training: Claude’s constitution instructs it to “basically never directly lie or actively deceive anyone it’s interacting with,” while OpenAI’s model spec says “Unless explicitly instructed to do so, the assistant must never lie or covertly pursue goals in a way that materially influences tool choices, content, or interaction patterns without disclosure and consent at the relevant authority level.” Yet they went ahead and misbehaved anyway.
It is possible that the models believed themselves to be in a simulation, not realizing that their actions were having real-world consequences. But some evidence suggests that this was not always the case: in the AISI tests, at one point the agent continued its malicious activity despite realizing it was on “real GitHub.” Even if they were confused, “they didn’t mean to do it” is cold comfort when something bad happens.
This week, Paul Christiano — until recently the head of AI safety at the US government’s Center for AI Standards and Innovation — wrote: “Modern AI systems are not aligned with human intent. We will likely train increasingly powerful models that take unintended actions in order to succeed at their task.”
The last few weeks show that this is the case. Meanwhile, policy solutions are still nonexistent: the voluntary White House AI framework reportedly does not require safety testing before internal deployments; state laws are beset by loopholes; and a federal law remains little more than an idea. Until governments do something, expect these first few stories of rogue AI behavior to be just the beginning.