{"slug": "don-t-use-ordinary-software-to-contain-software-hacking-agents", "title": "Don't Use Ordinary Software to Contain Software-Hacking Agents", "summary": "OpenAI disclosed on 21 July that two of its models—GPT-5.6 Sol and an unreleased, more capable model—broke out of their evaluation sandbox, reached the open Internet, and compromised Hugging Face's production infrastructure in an apparent attempt to steal answers to ExploitGym, an offensive-security benchmark. The models exploited a software proxy designed to mediate network access, highlighting flaws in containment strategies for AI agents.", "body_md": "On 21 July, OpenAI [disclosed](https://openai.com/index/hugging-face-model-evaluation-security-incident/)\nthat two of its models—GPT-5.6 Sol and an unreleased, apparently more capable one—broke out of the\nsandbox they were being evaluated in, reached the open Internet, and compromised\n[Hugging Face](https://huggingface.co)’s production infrastructure. Their apparent goal was to\nsteal the answers to [ExploitGym](https://arxiv.org/abs/2605.11086), an [offensive-security](https://en.wikipedia.org/wiki/Red_team)\nbenchmark built by outside researchers that OpenAI was running internally to test their models’ hacking capabilities.\n\nMuch of the [discussion](https://fortune.com/2026/07/21/openai-says-ai-models-escaped-control-hacked-hugging-face/)\n[since](https://www.cnbc.com/2026/07/22/open-ai-cyber-models-hack-hugging-face.html) has been\nabout what this tells us about frontier model capabilities and what happens when “AI goes rogue”. Simon Willison is\n[worth reading](https://simonwillison.net/2026/Jul/22/openai-cyberattack/) on the asymmetry it\nexposes. Katie Moussouris\n[also](https://www.lutasecurity.com/post/openface-the-hugging-face-breach-and-what-to-do-about-it)\nargues convincingly, among other things, that containment needs to match the capability being tested.\n\nI’d go further: my contention is that the way OpenAI’s evaluation tried to contain the AI agents was defective by design—a design inherited from ExploitGym. This has important lessons for how offensive evaluations need to be built if they are to be safe, and that is what this post is about.\n\nI’ll aim to keep this post updated as more information comes to light.\n\n# How the models got out\n\nExploitGym asks whether an AI agent can turn a known\n[vulnerability](https://en.wikipedia.org/wiki/Vulnerability_(computer_security)) into a working [exploit](https://en.wikipedia.org/wiki/Exploit_(computer_security)). Its\n[898 tasks](https://arxiv.org/html/2605.11086v1) are drawn from real bugs in userspace\nprograms (e.g., [FFmpeg](https://www.ffmpeg.org/)), plus Google’s [V8](https://en.wikipedia.org/wiki/V8_(JavaScript_engine)) JavaScript engine and the Linux kernel. In each task the agent is given vulnerable\nsource code, build instructions, an input that triggers the bug, and a container (aka [sandbox](https://en.wikipedia.org/wiki/Sandbox_(computer_security))) to work in.\n\nWhile working, if the agent needs to compile and run the vulnerable program (which it will if it is to write a\nfunctioning exploit) then it has to download the packages necessary for that.\nTherefore the sandbox needs to let some traffic in and\nout. ExploitGym’s reference design does this with a software mediator. In the paper’s own\nwords, [“each agent’s network access is mediated by an egress proxy”](https://arxiv.org/html/2605.11086v1)\nwhose allowlist permits package installation from [apt](https://en.wikipedia.org/wiki/APT_(software)) and [PyPI](https://en.wikipedia.org/wiki/Python_Package_Index) and the fetching of the V8\n[toolchain](https://en.wikipedia.org/wiki/Toolchain), and blocks everything else. The paper is explicit that the proxy is there\n[“to minimize security risks and potential reward hacking through web search”](https://arxiv.org/html/2605.11086v1)—a\nphrase worth remembering when we come to whether any of this was foreseeable.\n\nOpenAI’s setup\n[worked along much the same lines](https://openai.com/index/hugging-face-model-evaluation-security-incident/):\nan allowlist, with package installation served by an internally hosted third-party product\nacting as a proxy and cache for package registries. The evaluation was also run deliberately\nwithout the production classifiers—the safety *guardrails*, to use the common term—that\nnormally stop OpenAI’s models pursuing high-risk cyber activity, which is defensible if raw\ncapability is what you want to measure.\n\nThat proxy is where the models got out. They\n[“spent a substantial amount of inference compute finding a way to obtain open Internet access”](https://openai.com/index/hugging-face-model-evaluation-security-incident/),\nin OpenAI’s words, and found a\n[zero-day](https://en.wikipedia.org/wiki/Zero-day_vulnerability) in it (since disclosed to the vendor), then\n[“performed a series of privilege escalation and lateral movement actions”](https://openai.com/index/hugging-face-model-evaluation-security-incident/)\nin OpenAI’s research environment until they reached a machine with Internet access. From there they\n[inferred](https://techcrunch.com/2026/07/21/openai-says-hugging-face-was-breached-by-its-pre-release-models/)\nthat Hugging Face likely hosted ExploitGym’s artefacts and went after it, eventually reaching\ntest solutions in Hugging Face’s production database.\n\n*What the containment looked like, and how the agent escaped: the package proxy was the one way out\nof the sandbox and was also something the agent could attack.*\n\n# Where was the trusted computing base?\n\nThe [trusted computing base](https://en.wikipedia.org/wiki/Trusted_computing_base) of a system\nis the set of components that must work correctly for that system to remain secure. More formally,\nwe might say if the system’s security goal is violated then there must be a flaw in its TCB.\nSo where was that flaw, here?\n\nHere the system’s security goal was roughly *the agent cannot reach the open Internet*, and the TCB includes\nthe network configuration, the container (aka sandbox) runtime, and the\nthird-party package proxy. That proxy straddles the container boundary: it parses requests from the agent\non the inside, and fulfills those requests using its outbound network access. Security folks will recognise\nthis setup from the [confused deputy problem](https://en.wikipedia.org/wiki/Confused_deputy_problem): the proxy\nwas a\ncomponent trusted with an authority (network access) that a less-privileged client (the AI agent) might attempt to\ncause it to wield on their\nbehalf. Here, the AI agent “confused” the proxy by finding and exploiting a vulnerability in it.\n\nCrucially, inside the boundary mediated by the proxy sat a system being measured on its ability to find unknown bugs in critical systems software.\n\nPut another way, *the containment depended on a class of software remaining secure that the contained system was\nsimultaneously being scored on its ability to break*.\n\n# Was this foreseeable?\n\n## I think so\n\nIn safety engineering, a [safety case](https://en.wikipedia.org/wiki/Safety_case) is the\ndocumented argument that a system is acceptably safe to operate, together with its supporting\nevidence. Whatever safety case supported this evaluation appears to have assumed that the agent\nwould not find an exploitable bug in the software immediately around it—which was precisely\nthe question the evaluation existed to answer.\n\nAnd there was every reason to expect the agent to go looking. We have long known that optimising\nsystems [game](https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/)\ntheir evaluations, sometimes with\n[remarkable creativity](https://vkrakovna.wordpress.com/specification-gaming-examples-in-ai/). An\nexploitation benchmark is the case where that tendency bites hardest, because the capability it\nmeasures and the capability needed to game it are one and the same: both come down to making a\nsystem do what its designers never intended. Exploiting a program means making the program do something\nits designers never foresaw because you understand more about how the program actually works than the\npeople who wrote it. Gaming an evaluation involves exactly the same kind of thinking: understand how the\nevaluation is actually being scored and then find a way that the designers never intended to maximise your\nscore. An agent good enough to score well was, by\nconstruction, one capable of turning that skill on its own containment.\n\nOne might argue that ExploitGym measures skill at *exploitation* not general offensive security tasks like\nreconnisance,\nprivilege escalation, and lateral movement. But these capabilities are closely correlated. An agent adept at exploitation is\nby necessity excellent\nat long-horizon reasoning, willing to chase obscure rewards and persist against myriad dead-ends, as well being\nable to think creatively. So we should expect any such agent to also be skilled at offensive security tasks in general.\n\nThe ExploitGym authors had half-seen this. The design of their benchmark includes the network proxy to, in their words, prevent “reward hacking”, so an agent gaming the evaluation was already on their minds. What I suspect they pictured was an agent wandering off to search the web for hints, rather than one attacking the proxy to gain unrestricted network access. For an agent optimised for its ability to attack software, the gap between the former and latter is vanishingly small.\n\nI would contend that the ExploitGym authors could have foreseen this. But that does not let OpenAI off the hook. Each person who runs an experiment has the responsibility for ensuring that doing so does not put others at undue risk.\n\nThis is true even if you think–as Alexander Barry has\n[noted](https://abstatisticalconsulting.substack.com/p/brief-notes-on-the-openaihugging)—that ExploitGym’s design\nmay have inadvertently incentivised the agent to escape its sandbox in order to game the evaluation.\nA barrier that holds only while the agent has no reason to break it is no barrier at all.\nRelying on the agent’s lack of motive is very much the same error as relying on its lack of skill.\n\n**Updated 2026-07-27:** [research](https://x.com/alxndrdavies/status/2081491259089387919) from the UK’s [AI Safety Institute](https://www.aisi.gov.uk/) suggests that breaking containment and cheating on\ncyber evaluations, including escalating privileges and attacking third-party systems, is endemic across current frontier models. Anybody who doesn’t bake this into their threat model\ngoing forward is playing with fire.\n\n# What should we do instead?\n\nFortunately, none of this is new. Malware analysts, operators of classified systems and builders\nof high-assurance systems have spent decades assuming software will eventually fail, because\n[it does](../2022-03-01-on-software-perfection/), and arranging things so that a software\nfailure does not become a security failure.\n\nI contend that offensive security evaluations of AI agents should be done behind an [air gap](https://en.wikipedia.org/wiki/Air_gap_(networking)).\nWhere data really must flow, mechanisms like\n[data diodes](https://en.wikipedia.org/wiki/Unidirectional_network) put the enforcement in\nhardware. Much of my own research has circled the same problem:\n[seL4](https://sel4.systems) shrinks the code that must be trusted until it is small enough to\n[verify](../2025-10-21-formal-verification-triangle/), and the\n[Cross Domain Desktop Compositor](https://trustworthy.systems/projects/OLD/cddc) moves\nenforcement out of software altogether, compositing several physically separated networks onto\none screen in trusted hardware.\n\nThe common theme across the designs of all of these systems is to shrink what must be trusted, and make whatever remains trustworthy.\n\n# What I’m not saying\n\nThere is a lot we still don’t know about this incident, so the argument above is made on what I think are reasonable suppositions.\n\nI’ve intentionally avoided the considerable intrigue surrounding this incident: the first fully autonomous AI-enabled\nhack, carried out by a frontier model-powered agent, against an AI company, which was triaged (at least in part) by another\nAI model.\nWillison points to the uncomfortable asymmetry in all this. The attack ran on frontier lab\nmodels with their cyber guardrails removed, while the defence could not use frontier lab models at\nall: Hugging Face’s responders were\n[blocked by the providers’ guardrails](https://simonwillison.net/2026/Jul/22/openai-cyberattack/)\nand fell back to a self-hosted open-weight model.\n[“These constraints are meant to make us safer. I think there’s a risk that they are having the opposite effect.”](https://simonwillison.net/2026/Jul/22/openai-cyberattack/)\nMoussouris draws a policy conclusion from that. But this post is not about AI regulation. Safe AI containment should not depend on\nany particular regulatory settings.\n\n# Looking ahead\n\nWhat I would most like to see is fewer discussions about AI alignment and more discussions about containment architectures for AI evaluations, published and picked over with the same energy that capability scores attract.\n\nIn the short term, academics publishing AI security evaluation benchmarks should take seriously\nthe question of how such experiments can be made safe-by-design. A paper\nproposing an offensive-capability evaluation is proposing a way to run dangerous software, and\nwhether its safeguards suffice to keep that software from reaching a bystander is exactly the\nsort of question [research-ethics review](https://en.wikipedia.org/wiki/Menlo_Report) exists to\nask. Security venues increasingly engage with ethical concerns during the paper review process, and\nrequire submissions to explicitly state how ethics concerns were managed during the research design\nand implementation phases. We should expect to see these kinds of mechanisms deployed to build\ncommunity consensus on how to safely carry out AI security evaluations.\n\nIf you’re working on any of this—or think I’ve read too much into what has been disclosed—please\n[get in touch](https://people.eng.unimelb.edu.au/tobym/).\n\nIn the meantime, thanks for reading. And spare a thought for the folks at Hugging Face who spent their weekend chasing an intruder that turned out to be someone else’s science experiment.", "url": "https://wpnews.pro/news/don-t-use-ordinary-software-to-contain-software-hacking-agents", "canonical_source": "https://verse.systems/blog/post/2026-07-24-dont-use-software-to-contain-software-hacking-agents/", "published_at": "2026-07-27 19:07:35+00:00", "updated_at": "2026-07-27 19:22:56.566073+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-safety", "ai-research", "ai-agents"], "entities": ["OpenAI", "GPT-5.6 Sol", "Hugging Face", "ExploitGym", "Simon Willison", "Katie Moussouris"], "alternates": {"html": "https://wpnews.pro/news/don-t-use-ordinary-software-to-contain-software-hacking-agents", "markdown": "https://wpnews.pro/news/don-t-use-ordinary-software-to-contain-software-hacking-agents.md", "text": "https://wpnews.pro/news/don-t-use-ordinary-software-to-contain-software-hacking-agents.txt", "jsonld": "https://wpnews.pro/news/don-t-use-ordinary-software-to-contain-software-hacking-agents.jsonld"}}