Jot Remember the security incident from Hugging Face I mentioned in the asymmetry problem? Turns out the “autonomous AI agent” responsible for the attack wasn't coming from some random cybercriminals but some OpenAI models running wild, trying to cheat their way into higher benchmark score:
Last week, Hugging Face[disclosed a new kind of security incident(opens in a new window)]after theydetected and contained an AI agent that compromised their infrastructure, something we expect to become more commonplace with the proliferation of increasingly cyber-capable models. After investigating,we now know that this particular incident was driven by a combination of OpenAI models— including GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes — while being internally tested on a[benchmark(opens in a new window)]of cyber capabilities.
You might wonder: How could OpenAI be so careless in their testing? Don't they isolate their test environment? It turns out they do, but not well enough for GPT-5.6 Sol to escape its jail:
Our benchmarks run in a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries.
[...]To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy.
With this access, our modelsperformed a series of privilege escalation and lateral movement actionsin our research testing environment until the models reached a node with Internet access.
Why Hugging Face? So it could find the answers of the Cybersecurity benchmark it was being tested on:
After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation. In one example,the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code executionpath on the Hugging Face servers.
This impressive feat set aside, we can see a trend on SOTA model where, as they get better for long horizon tasks, become harder to keep on leash.
This isn't the first time a model go wild during training. One of the first publicly documented occurrence was in a paper from Alibaba's AI team:
Early one morning, our team was urgently convened after Alibaba Cloud’s managed firewall flagged a burst of security-policy violations originating from our training servers. The alerts were severe and heterogeneous, including attempts to probe or access internal-network resources and traffic patterns consistent with cryptomining-related activity. [...]
We then correlated firewall timestamps with our system telemetry and RL traces, and found that the anomalous outbound traffic consistently coincided with specific episodes in which the agent invoked tools and executed code. [...]
Crucially, these behaviors were not requested by the task prompts and were not required for task completion under the intended sandbox constraints. [...]In the most striking instance, the agent established and used a reverse SSH tunnel from an Alibaba Cloud instance to an external IP address—an outbound-initiated remote access channel that can effectively neutralize ingress filtering and erode supervisory control.We also observed the unauthorized repurposing of provisioned GPU capacity for cryptocurrency mining, quietly diverting compute away from training, inflating operational costs, and introducing clear legal and reputational exposure. Notably, these events were not triggered by prompts requesting tunneling or mining; instead, they emerged asinstrumental side effectsof autonomous tool use under RL optimization.
This was also not the first time 5.6 Sol got flagged for cheating its way into benchmarks. METR mentioned it back in June: We initiated an evaluation of GPT-5.6 Sol on our Time Horizon 1.1 suite of software tasks. However, the resulting measurement depends heavily on our detection and treatment of cheating attempts by the model, andGPT-5.6 Sol’s detected cheating rate was higher than any public model we have evaluated on our ReAct agent harness.
For our task suite, we define “cheating” as behavior where the model improves evaluation performance by exploiting bugs in the evaluation environment or by adopting strategies disallowed by the task, rather than solving the task within the expected evaluation constraints. Some examples we saw when evaluating GPT-5.6 Solincluded the model packaging exploits in its intermediate submissions to reveal information about a task’s hidden test suite and, in another task, extracting hidden source code detailing the expected answer.
This was not news for OpenAI:
We noted from our observations and incidents that OpenAI shared with us that the model had some overt undesirable propensities, including cheating and concealing misbehavior.
The GPT-5.6 System card itself explicitly mention the model tendency to misalign:
We find that GPT-5.6 Sol, more often than its predecessor, can be overly persistent in pursuing user goals, to the point oftaking actions that go beyond what the user intended.[...]
This manifests as the model being overly agentic in circumventing restrictions it faces when attempting the requested task, being careless in taking actions which may be destructive beyond the scope of the task, ordeceptive when reporting its results to users.[...]
Examples includedeleting data from cloud storagewithout requesting user approval,disabling monitoring systems, using obfuscation strategies toget around security controls, andup potentially sensitive data (such as code, credentials, images, or personal data) to unapproved services. [...]
We have observed instances of the model cheating on tasks and fabricating research results.
Something tells me we will have a fun time harnessing such models in corporate workload. Imagine having company secrets exfiltrated on unapproved services due to some agents misunderstanding their objectives or cheating for a quicker completion? Even more concerning, it's not clear how devastating those post-mythos models can be when put in the wrong hands.
A few weeks ago, CleverHans Lab released a paper on how AI Agents Enable Adaptive Computer Worms.
A computer worm is self-replicating malware that spreads across a network without human intervention. The WannaCry worm (2017) disrupted critical infrastructure across 150 countries by exploiting a single vulnerability.Traditional worms can be stopped by patching the specific vulnerability they exploit. Our adaptive worm cannot be stopped this way: it uses a recursive reasoning loop to detect and exploit diverse vulnerabilities as it propagates. [...]Each compromised machine becomes part of the worm’s own infrastructure, providing compute or reach for further attacks. [...]
We discovered that it is possible to create anAI-driven computer worm, using only small, free AI models, that can autonomously identify each machine’s unique weak points (including vulnerabilities just reported by industry and misconfigurations such as reused passwords) and exploit them, hijacking computing power to take over regular devices such as laptops, cameras and everything else online, and then copying itself onto servers and networks to either steal data or launch new attacks.
Now, this (undisclosed) worm was relying on a small, local model, incapable of finding zero-days on its own. Constrained by its size and struggling for proper compute power, the worm was also quite slow, taking five days to compromise half their test lab. It doesn't require much effort to imagine a stronger variant which would rely on remote computing capabilities and bigger, unrestricted models like GLM 5.2 or Kimi K3. The model could also be smarter and spread quickly with prevalent vulnerabilities, like a standard worm would do, while cooking extra zero-days in parallel for when the victims start patching.
I still remember the Blaster worm, I would get my freshly installed computer compromised within 30 minutes of connecting it to the Internet. Now imagine a Wannacry or Blaster that never stops as the vulnerability it uses keeps changing.
What could we do against it? Besides the obvious security measures to reduce the attack surface, it feels like to keep up the pace of AI attackers, we will need to unleash AI defenders. Yet, our salvation is unlikely to come from the American SOTA models. After OpenAI and Anthropics, it's now Google's turn to gate keep their latest “cyber” variant:
Given the dual-use nature of this technology, we have taken an intentional approach to deploying 3.5 Flash Cyber.The model will be exclusively available to governments and trusted partnersvia[CodeMender]soon as part of a limited-access pilot program. This will give frontline defenders a head start in finding and fixing critical vulnerabilities before they can be exploited, while mitigating against broader misuse.
Meanwhile, Chinese models like GLM, Kimi or Qwen are getting extremely close to their US counterpart, with none of their cybersecurity limitations. To stay in the game, I expect European companies will need to start leveraging inference providers supporting such models.
Losing the Internet might just be a few misalignments away.