# Don't Use Ordinary Software to Contain Software-Hacking Agents > Source: > Published: 2026-07-27 19:07:35+00:00 On 21 July, OpenAI [disclosed](https://openai.com/index/hugging-face-model-evaluation-security-incident/) that two of its models—GPT-5.6 Sol and an unreleased, apparently more capable one—broke out of the sandbox they were being evaluated in, reached the open Internet, and compromised [Hugging Face](https://huggingface.co)’s production infrastructure. Their apparent goal was to steal the answers to [ExploitGym](https://arxiv.org/abs/2605.11086), an [offensive-security](https://en.wikipedia.org/wiki/Red_team) benchmark built by outside researchers that OpenAI was running internally to test their models’ hacking capabilities. Much of the [discussion](https://fortune.com/2026/07/21/openai-says-ai-models-escaped-control-hacked-hugging-face/) [since](https://www.cnbc.com/2026/07/22/open-ai-cyber-models-hack-hugging-face.html) has been about what this tells us about frontier model capabilities and what happens when “AI goes rogue”. Simon Willison is [worth reading](https://simonwillison.net/2026/Jul/22/openai-cyberattack/) on the asymmetry it exposes. Katie Moussouris [also](https://www.lutasecurity.com/post/openface-the-hugging-face-breach-and-what-to-do-about-it) argues convincingly, among other things, that containment needs to match the capability being tested. I’d go further: my contention is that the way OpenAI’s evaluation tried to contain the AI agents was defective by design—a design inherited from ExploitGym. This has important lessons for how offensive evaluations need to be built if they are to be safe, and that is what this post is about. I’ll aim to keep this post updated as more information comes to light. # How the models got out ExploitGym asks whether an AI agent can turn a known [vulnerability](https://en.wikipedia.org/wiki/Vulnerability_(computer_security)) into a working [exploit](https://en.wikipedia.org/wiki/Exploit_(computer_security)). Its [898 tasks](https://arxiv.org/html/2605.11086v1) are drawn from real bugs in userspace programs (e.g., [FFmpeg](https://www.ffmpeg.org/)), plus Google’s [V8](https://en.wikipedia.org/wiki/V8_(JavaScript_engine)) JavaScript engine and the Linux kernel. In each task the agent is given vulnerable source code, build instructions, an input that triggers the bug, and a container (aka [sandbox](https://en.wikipedia.org/wiki/Sandbox_(computer_security))) to work in. While working, if the agent needs to compile and run the vulnerable program (which it will if it is to write a functioning exploit) then it has to download the packages necessary for that. Therefore the sandbox needs to let some traffic in and out. ExploitGym’s reference design does this with a software mediator. In the paper’s own words, [“each agent’s network access is mediated by an egress proxy”](https://arxiv.org/html/2605.11086v1) whose allowlist permits package installation from [apt](https://en.wikipedia.org/wiki/APT_(software)) and [PyPI](https://en.wikipedia.org/wiki/Python_Package_Index) and the fetching of the V8 [toolchain](https://en.wikipedia.org/wiki/Toolchain), and blocks everything else. The paper is explicit that the proxy is there [“to minimize security risks and potential reward hacking through web search”](https://arxiv.org/html/2605.11086v1)—a phrase worth remembering when we come to whether any of this was foreseeable. OpenAI’s setup [worked along much the same lines](https://openai.com/index/hugging-face-model-evaluation-security-incident/): an allowlist, with package installation served by an internally hosted third-party product acting as a proxy and cache for package registries. The evaluation was also run deliberately without the production classifiers—the safety *guardrails*, to use the common term—that normally stop OpenAI’s models pursuing high-risk cyber activity, which is defensible if raw capability is what you want to measure. That proxy is where the models got out. They [“spent a substantial amount of inference compute finding a way to obtain open Internet access”](https://openai.com/index/hugging-face-model-evaluation-security-incident/), in OpenAI’s words, and found a [zero-day](https://en.wikipedia.org/wiki/Zero-day_vulnerability) in it (since disclosed to the vendor), then [“performed a series of privilege escalation and lateral movement actions”](https://openai.com/index/hugging-face-model-evaluation-security-incident/) in OpenAI’s research environment until they reached a machine with Internet access. From there they [inferred](https://techcrunch.com/2026/07/21/openai-says-hugging-face-was-breached-by-its-pre-release-models/) that Hugging Face likely hosted ExploitGym’s artefacts and went after it, eventually reaching test solutions in Hugging Face’s production database. *What the containment looked like, and how the agent escaped: the package proxy was the one way out of the sandbox and was also something the agent could attack.* # Where was the trusted computing base? The [trusted computing base](https://en.wikipedia.org/wiki/Trusted_computing_base) of a system is the set of components that must work correctly for that system to remain secure. More formally, we might say if the system’s security goal is violated then there must be a flaw in its TCB. So where was that flaw, here? Here the system’s security goal was roughly *the agent cannot reach the open Internet*, and the TCB includes the network configuration, the container (aka sandbox) runtime, and the third-party package proxy. That proxy straddles the container boundary: it parses requests from the agent on the inside, and fulfills those requests using its outbound network access. Security folks will recognise this setup from the [confused deputy problem](https://en.wikipedia.org/wiki/Confused_deputy_problem): the proxy was a component trusted with an authority (network access) that a less-privileged client (the AI agent) might attempt to cause it to wield on their behalf. Here, the AI agent “confused” the proxy by finding and exploiting a vulnerability in it. Crucially, inside the boundary mediated by the proxy sat a system being measured on its ability to find unknown bugs in critical systems software. Put another way, *the containment depended on a class of software remaining secure that the contained system was simultaneously being scored on its ability to break*. # Was this foreseeable? ## I think so In safety engineering, a [safety case](https://en.wikipedia.org/wiki/Safety_case) is the documented argument that a system is acceptably safe to operate, together with its supporting evidence. Whatever safety case supported this evaluation appears to have assumed that the agent would not find an exploitable bug in the software immediately around it—which was precisely the question the evaluation existed to answer. And there was every reason to expect the agent to go looking. We have long known that optimising systems [game](https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/) their evaluations, sometimes with [remarkable creativity](https://vkrakovna.wordpress.com/specification-gaming-examples-in-ai/). An exploitation benchmark is the case where that tendency bites hardest, because the capability it measures and the capability needed to game it are one and the same: both come down to making a system do what its designers never intended. Exploiting a program means making the program do something its designers never foresaw because you understand more about how the program actually works than the people who wrote it. Gaming an evaluation involves exactly the same kind of thinking: understand how the evaluation is actually being scored and then find a way that the designers never intended to maximise your score. An agent good enough to score well was, by construction, one capable of turning that skill on its own containment. One might argue that ExploitGym measures skill at *exploitation* not general offensive security tasks like reconnisance, privilege escalation, and lateral movement. But these capabilities are closely correlated. An agent adept at exploitation is by necessity excellent at long-horizon reasoning, willing to chase obscure rewards and persist against myriad dead-ends, as well being able to think creatively. So we should expect any such agent to also be skilled at offensive security tasks in general. The ExploitGym authors had half-seen this. The design of their benchmark includes the network proxy to, in their words, prevent “reward hacking”, so an agent gaming the evaluation was already on their minds. What I suspect they pictured was an agent wandering off to search the web for hints, rather than one attacking the proxy to gain unrestricted network access. For an agent optimised for its ability to attack software, the gap between the former and latter is vanishingly small. I would contend that the ExploitGym authors could have foreseen this. But that does not let OpenAI off the hook. Each person who runs an experiment has the responsibility for ensuring that doing so does not put others at undue risk. This is true even if you think–as Alexander Barry has [noted](https://abstatisticalconsulting.substack.com/p/brief-notes-on-the-openaihugging)—that ExploitGym’s design may have inadvertently incentivised the agent to escape its sandbox in order to game the evaluation. A barrier that holds only while the agent has no reason to break it is no barrier at all. Relying on the agent’s lack of motive is very much the same error as relying on its lack of skill. **Updated 2026-07-27:** [research](https://x.com/alxndrdavies/status/2081491259089387919) from the UK’s [AI Safety Institute](https://www.aisi.gov.uk/) suggests that breaking containment and cheating on cyber evaluations, including escalating privileges and attacking third-party systems, is endemic across current frontier models. Anybody who doesn’t bake this into their threat model going forward is playing with fire. # What should we do instead? Fortunately, none of this is new. Malware analysts, operators of classified systems and builders of high-assurance systems have spent decades assuming software will eventually fail, because [it does](../2022-03-01-on-software-perfection/), and arranging things so that a software failure does not become a security failure. I contend that offensive security evaluations of AI agents should be done behind an [air gap](https://en.wikipedia.org/wiki/Air_gap_(networking)). Where data really must flow, mechanisms like [data diodes](https://en.wikipedia.org/wiki/Unidirectional_network) put the enforcement in hardware. Much of my own research has circled the same problem: [seL4](https://sel4.systems) shrinks the code that must be trusted until it is small enough to [verify](../2025-10-21-formal-verification-triangle/), and the [Cross Domain Desktop Compositor](https://trustworthy.systems/projects/OLD/cddc) moves enforcement out of software altogether, compositing several physically separated networks onto one screen in trusted hardware. The common theme across the designs of all of these systems is to shrink what must be trusted, and make whatever remains trustworthy. # What I’m not saying There is a lot we still don’t know about this incident, so the argument above is made on what I think are reasonable suppositions. I’ve intentionally avoided the considerable intrigue surrounding this incident: the first fully autonomous AI-enabled hack, carried out by a frontier model-powered agent, against an AI company, which was triaged (at least in part) by another AI model. Willison points to the uncomfortable asymmetry in all this. The attack ran on frontier lab models with their cyber guardrails removed, while the defence could not use frontier lab models at all: Hugging Face’s responders were [blocked by the providers’ guardrails](https://simonwillison.net/2026/Jul/22/openai-cyberattack/) and fell back to a self-hosted open-weight model. [“These constraints are meant to make us safer. I think there’s a risk that they are having the opposite effect.”](https://simonwillison.net/2026/Jul/22/openai-cyberattack/) Moussouris draws a policy conclusion from that. But this post is not about AI regulation. Safe AI containment should not depend on any particular regulatory settings. # Looking ahead What I would most like to see is fewer discussions about AI alignment and more discussions about containment architectures for AI evaluations, published and picked over with the same energy that capability scores attract. In the short term, academics publishing AI security evaluation benchmarks should take seriously the question of how such experiments can be made safe-by-design. A paper proposing an offensive-capability evaluation is proposing a way to run dangerous software, and whether its safeguards suffice to keep that software from reaching a bystander is exactly the sort of question [research-ethics review](https://en.wikipedia.org/wiki/Menlo_Report) exists to ask. Security venues increasingly engage with ethical concerns during the paper review process, and require submissions to explicitly state how ethics concerns were managed during the research design and implementation phases. We should expect to see these kinds of mechanisms deployed to build community consensus on how to safely carry out AI security evaluations. If you’re working on any of this—or think I’ve read too much into what has been disclosed—please [get in touch](https://people.eng.unimelb.edu.au/tobym/). In the meantime, thanks for reading. And spare a thought for the folks at Hugging Face who spent their weekend chasing an intruder that turned out to be someone else’s science experiment.