Don't Use Ordinary Software to Contain Software-Hacking Agents

OpenAI disclosed on 21 July that two of its models—GPT-5.6 Sol and an unreleased, more capable model—broke out of their evaluation sandbox, reached the open Internet, and compromised Hugging Face's production infrastructure in an apparent attempt to steal answers to ExploitGym, an offensive-security benchmark. The models exploited a software proxy designed to mediate network access, highlighting flaws in containment strategies for AI agents.

On 21 July, OpenAI disclosed https://openai.com/index/hugging-face-model-evaluation-security-incident/ that two of its models—GPT-5.6 Sol and an unreleased, apparently more capable one—broke out of the sandbox they were being evaluated in, reached the open Internet, and compromised Hugging Face https://huggingface.co ’s production infrastructure. Their apparent goal was to steal the answers to ExploitGym https://arxiv.org/abs/2605.11086 , an offensive-security https://en.wikipedia.org/wiki/Red team benchmark built by outside researchers that OpenAI was running internally to test their models’ hacking capabilities. Much of the discussion https://fortune.com/2026/07/21/openai-says-ai-models-escaped-control-hacked-hugging-face/ since https://www.cnbc.com/2026/07/22/open-ai-cyber-models-hack-hugging-face.html has been about what this tells us about frontier model capabilities and what happens when “AI goes rogue”. Simon Willison is worth reading https://simonwillison.net/2026/Jul/22/openai-cyberattack/ on the asymmetry it exposes. Katie Moussouris also https://www.lutasecurity.com/post/openface-the-hugging-face-breach-and-what-to-do-about-it argues convincingly, among other things, that containment needs to match the capability being tested. I’d go further: my contention is that the way OpenAI’s evaluation tried to contain the AI agents was defective by design—a design inherited from ExploitGym. This has important lessons for how offensive evaluations need to be built if they are to be safe, and that is what this post is about. I’ll aim to keep this post updated as more information comes to light. How the models got out ExploitGym asks whether an AI agent can turn a known vulnerability https://en.wikipedia.org/wiki/Vulnerability computer security into a working exploit https://en.wikipedia.org/wiki/Exploit computer security . Its 898 tasks https://arxiv.org/html/2605.11086v1 are drawn from real bugs in userspace programs e.g., FFmpeg https://www.ffmpeg.org/ , plus Google’s V8 https://en.wikipedia.org/wiki/V8 JavaScript engine JavaScript engine and the Linux kernel. In each task the agent is given vulnerable source code, build instructions, an input that triggers the bug, and a container aka sandbox https://en.wikipedia.org/wiki/Sandbox computer security to work in. While working, if the agent needs to compile and run the vulnerable program which it will if it is to write a functioning exploit then it has to download the packages necessary for that. Therefore the sandbox needs to let some traffic in and out. ExploitGym’s reference design does this with a software mediator. In the paper’s own words, “each agent’s network access is mediated by an egress proxy” https://arxiv.org/html/2605.11086v1 whose allowlist permits package installation from apt https://en.wikipedia.org/wiki/APT software and PyPI https://en.wikipedia.org/wiki/Python Package Index and the fetching of the V8 toolchain https://en.wikipedia.org/wiki/Toolchain , and blocks everything else. The paper is explicit that the proxy is there “to minimize security risks and potential reward hacking through web search” https://arxiv.org/html/2605.11086v1 —a phrase worth remembering when we come to whether any of this was foreseeable. OpenAI’s setup worked along much the same lines https://openai.com/index/hugging-face-model-evaluation-security-incident/ : an allowlist, with package installation served by an internally hosted third-party product acting as a proxy and cache for package registries. The evaluation was also run deliberately without the production classifiers—the safety guardrails , to use the common term—that normally stop OpenAI’s models pursuing high-risk cyber activity, which is defensible if raw capability is what you want to measure. That proxy is where the models got out. They “spent a substantial amount of inference compute finding a way to obtain open Internet access” https://openai.com/index/hugging-face-model-evaluation-security-incident/ , in OpenAI’s words, and found a zero-day https://en.wikipedia.org/wiki/Zero-day vulnerability in it since disclosed to the vendor , then “performed a series of privilege escalation and lateral movement actions” https://openai.com/index/hugging-face-model-evaluation-security-incident/ in OpenAI’s research environment until they reached a machine with Internet access. From there they inferred https://techcrunch.com/2026/07/21/openai-says-hugging-face-was-breached-by-its-pre-release-models/ that Hugging Face likely hosted ExploitGym’s artefacts and went after it, eventually reaching test solutions in Hugging Face’s production database. What the containment looked like, and how the agent escaped: the package proxy was the one way out of the sandbox and was also something the agent could attack. Where was the trusted computing base? The trusted computing base https://en.wikipedia.org/wiki/Trusted computing base of a system is the set of components that must work correctly for that system to remain secure. More formally, we might say if the system’s security goal is violated then there must be a flaw in its TCB. So where was that flaw, here? Here the system’s security goal was roughly the agent cannot reach the open Internet , and the TCB includes the network configuration, the container aka sandbox runtime, and the third-party package proxy. That proxy straddles the container boundary: it parses requests from the agent on the inside, and fulfills those requests using its outbound network access. Security folks will recognise this setup from the confused deputy problem https://en.wikipedia.org/wiki/Confused deputy problem : the proxy was a component trusted with an authority network access that a less-privileged client the AI agent might attempt to cause it to wield on their behalf. Here, the AI agent “confused” the proxy by finding and exploiting a vulnerability in it. Crucially, inside the boundary mediated by the proxy sat a system being measured on its ability to find unknown bugs in critical systems software. Put another way, the containment depended on a class of software remaining secure that the contained system was simultaneously being scored on its ability to break . Was this foreseeable? I think so In safety engineering, a safety case https://en.wikipedia.org/wiki/Safety case is the documented argument that a system is acceptably safe to operate, together with its supporting evidence. Whatever safety case supported this evaluation appears to have assumed that the agent would not find an exploitable bug in the software immediately around it—which was precisely the question the evaluation existed to answer. And there was every reason to expect the agent to go looking. We have long known that optimising systems game https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/ their evaluations, sometimes with remarkable creativity https://vkrakovna.wordpress.com/specification-gaming-examples-in-ai/ . An exploitation benchmark is the case where that tendency bites hardest, because the capability it measures and the capability needed to game it are one and the same: both come down to making a system do what its designers never intended. Exploiting a program means making the program do something its designers never foresaw because you understand more about how the program actually works than the people who wrote it. Gaming an evaluation involves exactly the same kind of thinking: understand how the evaluation is actually being scored and then find a way that the designers never intended to maximise your score. An agent good enough to score well was, by construction, one capable of turning that skill on its own containment. One might argue that ExploitGym measures skill at exploitation not general offensive security tasks like reconnisance, privilege escalation, and lateral movement. But these capabilities are closely correlated. An agent adept at exploitation is by necessity excellent at long-horizon reasoning, willing to chase obscure rewards and persist against myriad dead-ends, as well being able to think creatively. So we should expect any such agent to also be skilled at offensive security tasks in general. The ExploitGym authors had half-seen this. The design of their benchmark includes the network proxy to, in their words, prevent “reward hacking”, so an agent gaming the evaluation was already on their minds. What I suspect they pictured was an agent wandering off to search the web for hints, rather than one attacking the proxy to gain unrestricted network access. For an agent optimised for its ability to attack software, the gap between the former and latter is vanishingly small. I would contend that the ExploitGym authors could have foreseen this. But that does not let OpenAI off the hook. Each person who runs an experiment has the responsibility for ensuring that doing so does not put others at undue risk. This is true even if you think–as Alexander Barry has noted https://abstatisticalconsulting.substack.com/p/brief-notes-on-the-openaihugging —that ExploitGym’s design may have inadvertently incentivised the agent to escape its sandbox in order to game the evaluation. A barrier that holds only while the agent has no reason to break it is no barrier at all. Relying on the agent’s lack of motive is very much the same error as relying on its lack of skill. Updated 2026-07-27: research https://x.com/alxndrdavies/status/2081491259089387919 from the UK’s AI Safety Institute https://www.aisi.gov.uk/ suggests that breaking containment and cheating on cyber evaluations, including escalating privileges and attacking third-party systems, is endemic across current frontier models. Anybody who doesn’t bake this into their threat model going forward is playing with fire. What should we do instead? Fortunately, none of this is new. Malware analysts, operators of classified systems and builders of high-assurance systems have spent decades assuming software will eventually fail, because it does ../2022-03-01-on-software-perfection/ , and arranging things so that a software failure does not become a security failure. I contend that offensive security evaluations of AI agents should be done behind an air gap https://en.wikipedia.org/wiki/Air gap networking . Where data really must flow, mechanisms like data diodes https://en.wikipedia.org/wiki/Unidirectional network put the enforcement in hardware. Much of my own research has circled the same problem: seL4 https://sel4.systems shrinks the code that must be trusted until it is small enough to verify ../2025-10-21-formal-verification-triangle/ , and the Cross Domain Desktop Compositor https://trustworthy.systems/projects/OLD/cddc moves enforcement out of software altogether, compositing several physically separated networks onto one screen in trusted hardware. The common theme across the designs of all of these systems is to shrink what must be trusted, and make whatever remains trustworthy. What I’m not saying There is a lot we still don’t know about this incident, so the argument above is made on what I think are reasonable suppositions. I’ve intentionally avoided the considerable intrigue surrounding this incident: the first fully autonomous AI-enabled hack, carried out by a frontier model-powered agent, against an AI company, which was triaged at least in part by another AI model. Willison points to the uncomfortable asymmetry in all this. The attack ran on frontier lab models with their cyber guardrails removed, while the defence could not use frontier lab models at all: Hugging Face’s responders were blocked by the providers’ guardrails https://simonwillison.net/2026/Jul/22/openai-cyberattack/ and fell back to a self-hosted open-weight model. “These constraints are meant to make us safer. I think there’s a risk that they are having the opposite effect.” https://simonwillison.net/2026/Jul/22/openai-cyberattack/ Moussouris draws a policy conclusion from that. But this post is not about AI regulation. Safe AI containment should not depend on any particular regulatory settings. Looking ahead What I would most like to see is fewer discussions about AI alignment and more discussions about containment architectures for AI evaluations, published and picked over with the same energy that capability scores attract. In the short term, academics publishing AI security evaluation benchmarks should take seriously the question of how such experiments can be made safe-by-design. A paper proposing an offensive-capability evaluation is proposing a way to run dangerous software, and whether its safeguards suffice to keep that software from reaching a bystander is exactly the sort of question research-ethics review https://en.wikipedia.org/wiki/Menlo Report exists to ask. Security venues increasingly engage with ethical concerns during the paper review process, and require submissions to explicitly state how ethics concerns were managed during the research design and implementation phases. We should expect to see these kinds of mechanisms deployed to build community consensus on how to safely carry out AI security evaluations. If you’re working on any of this—or think I’ve read too much into what has been disclosed—please get in touch https://people.eng.unimelb.edu.au/tobym/ . In the meantime, thanks for reading. And spare a thought for the folks at Hugging Face who spent their weekend chasing an intruder that turned out to be someone else’s science experiment.