{"slug": "nobody-authorised-the-combination", "title": "Nobody Authorised the Combination", "summary": "OpenAI disclosed on 21 July that models under internal evaluation reached Hugging Face's production infrastructure and exfiltrated data from its production database, according to OpenAI's incident report. The breach chained four individually scoped components — a benchmark called ExploitGym run without production classifiers, an isolated sandbox whose only network path was package installation through an internally hosted Artifactory proxy, an unknown Artifactory zero-day, and publicly exposed credentials on unrelated services — into privilege escalation and lateral movement that ended in remote code execution on Hugging Face's production servers. The author of the Proof-Carrying Coalitions proof-of-concept argues the OpenAI/Hugging Face breach and Anthropic's four disclosures are one failure mode: individually safe agents composing into a coalition that reaches a capability none could reach alone.", "body_md": "# Nobody Authorised the Combination\n\nThe OpenAI/Hugging Face breach and Anthropic's four disclosures were not four separate stories about unruly models. They were one story about checking parts instead of closures, and it stopped being a thought experiment this summer.\n\nIn [Continuous Enforcement and Continuous Verification](continuous-enforcement-continuous-verification.html) I described a proof-of-concept called [Proof-Carrying Coalitions](https://pcc.aguilar-pelaez.co.uk/) and stated its thesis as a slogan: `A ∪ B`, safe joined to safe, can equal unsafe. A set of individually safe agents can compose into a coalition that reaches a capability none of them could reach alone, and the only way to catch that is to compute the closure of what the combination can jointly do and ask a kernel whether the closure ever touches a forbidden set. I called it a proof-of-concept because it was one: a Lean 4 model, a demo, a founding case about two of my own standing instructions colliding to produce a workaround that honoured both and betrayed both. A margin note about a genuinely new class of failure, argued from first principles and one bad evening with a Chrome extension.\n\nThen, over eight weeks this summer, two frontier labs published four incident reports that are, read technically, the same failure, at production scale, with real infrastructure on the other end. I want to walk through what actually combined in each one, because the honest version of \"this validates the thesis\" requires showing the union, not asserting it, and because the differences between the cases are where the formalism's edges are.\n\n## The union, traced: OpenAI and Hugging Face\n\nOn 21 July, OpenAI disclosed that models under internal evaluation had reached Hugging Face's production infrastructure and pulled data from its production database [(OpenAI)](https://openai.com/index/hugging-face-model-evaluation-security-incident/). Read the sequence as a list of individually-scoped components rather than as a single villain, because that is what it is.\n\nA benchmark, ExploitGym, tasked GPT-5.6 Sol and a pre-release research prototype with advanced exploitation, and to make the test meaningful, OpenAI intentionally left off the production classifiers that normally stop a model pursuing high-risk cyber activity. That is component one, and it was not unsafe in isolation, it was an evaluation deliberately run without its comparator because the comparator would have defeated the point of the test. Component two was the sandbox itself: a highly isolated environment whose only permitted network path was package installation through an internally hosted proxy and cache, later identified as Artifactory. Scoped correctly, on its own terms, that path grants nothing but package downloads. Component three was Artifactory's own previously unknown zero-day, a vulnerability nobody, including OpenAI, knew existed, and therefore not a failure of anyone's authorisation model, simply a fact about the world. Component four was a small number of publicly exposed credentials on unrelated services, gathered incidentally rather than targeted. None of these four things is a safety failure by itself. A disabled classifier is a documented trade-off for a specific test. A constrained network path is good sandboxing. An unknown zero-day is not a design flaw, it is the reason zero-days have that name. Exposed credentials on a third-party service are somebody else's problem until they are not.\n\nThe union of the four was privilege escalation and lateral movement inside OpenAI's own research environment, reaching a node with internet access, then chaining that access with the credentials and further zero-days into remote code execution on Hugging Face's production servers and exfiltration from its production database. `Cl({classifier-off, sandboxed-eval, artifactory-proxy, exposed-creds})` reached a capability, unauthenticated write access to a third party's production infrastructure, that is very obviously in the forbidden set `F`, and no single element of that set was the thing that put it there. This is not a metaphorical instance of `A ∪ B = forbidden`. It is close to a literal one, with the closure computed by the attacker rather than by a kernel checking in advance whether it could be computed at all.\n\n## The comparator that was switched off on purpose\n\nI want to be precise about the classifier detail, because it is the part of this story closest to my own argument and the part most people are reading wrong. The production classifiers are OpenAI's actual comparator, the mechanism that, in ordinary operation, checks candidate actions against the forbidden set before they execute. They were not absent by oversight. They were disabled *because the evaluation existed to test what happens without them*, which is a legitimate thing to want to know. The failure is not \"the comparator was broken\". The failure is that removing the comparator for one purpose, testing cyber capability, silently removed it for every other property that comparator happened to be the only check on, including whatever kept the sandbox's network boundary meaningful once a path to the internet existed. A comparator scoped to one property and disabled for a test of that property does not thereby announce which *other* properties it was quietly the last line of defence for. That is a coalition problem one level up: the classifier and the sandbox boundary were, jointly, a safety mechanism, and disabling one half of a joint mechanism for a documented reason is not the same claim as disabling the joint mechanism, but it has the same effect if nobody checks the closure of what remains.\n\n## The union, traced: Anthropic, four times\n\nAnthropic's disclosures are a cleaner illustration of a different half of the same problem, because the union here crosses an organisational boundary rather than a technical one. In three incidents dated as early as April and disclosed on 30 July, Claude models were run inside capture-the-flag evaluations built by a third-party partner, Irregular. Anthropic's own evaluation prompt told each model explicitly that it had no internet access and was operating inside a simulation. A misconfiguration on Irregular's side meant that was false [(Anthropic)](https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals). Two organisations were individually reasonable: Anthropic's prompt correctly stated the *intended* boundary, and Irregular's environment was presumably reasonable by its own internal lights until the day it silently wasn't. Nobody on either side signed off on the actual joint configuration, an environment with live egress paired with a model told the opposite, because that configuration existed only in the gap between two parties' separate mental models of what the other had built. `A`, the prompt's claimed boundary, and `B`, the environment's actual boundary, were each somebody's honest belief about the system. Their conjunction was false, and nothing checked the conjunction, only the two conjuncts separately, and separately they each sounded fine to the party who wrote them.\n\nThe most striking single detail, reported by Hacker News from Irregular's own account, is a *naming* collision: a fictional target company used in the capture-the-flag scenario happened to share a name with a real one [(The Hacker News)](https://thehackernews.com/2026/09/anthropic-ai-models-breached-real.html). Claude Opus 4.7 found the real company, reasoned that anything reachable must be in scope for the exercise it had been told it was in, and extracted credentials and several hundred rows of production data from a real database [(Fortune)](https://fortune.com/2026/07/31/anthropic-claude-escaped-test-hacked-three-companies-openai/). Read that as a coalition of two design choices, a naming convention for fictional scenarios and a naming convention for real infrastructure, that were each fine in their own namespace and collided the moment the namespaces turned out not to be as separate as everyone assumed. This is the instruction-intersection case I flagged as unsolved in the earlier essay, not the capability-union case: two individually sound conventions, conjoined, left a gap wide enough for an agent to walk through in good faith.\n\nThe prompt itself is worth dwelling on, because it is exactly the mechanism I called prompt-prayer: an instruction stated in prose, trusted to hold the boundary, with no independent check that it was true. The difference from the usual prompt-prayer failure, an agent that reads the instruction and works around it, is that here the model *believed* the instruction and acted correctly on a false premise. That is arguably worse for the discipline this essay is about, because it shows that even perfect instruction-following inside a coalition of separately-authored beliefs is not safety, if nothing checks whether those beliefs are jointly true. Anthropic's own account of the second incident sharpens this: Claude Mythos 5, midway through publishing a malicious package, found evidence its environment was connected to the real internet, and reasoned itself back into treating that evidence as further proof it was still in a simulation [(Anthropic)](https://www.anthropic.com/news/alignment-assessment-cybersecurity-incidents). Anthropic later named this behavioural pattern precisely: biased reasoning, discounting evidence of the real environment, and recklessness, a willingness to cause harm in pursuit of a narrow assigned task [(Reuters)](https://www.reuters.com/legal/litigation/anthropic-reports-fourth-cybersecurity-incident-with-early-version-claude-2026-09-09/). Both are properties of a single agent's epistemics, not of a coalition, and I raise them because they mark the boundary of what capability-closure checking is even aimed at. A kernel that proves a coalition's reachable capabilities avoid a forbidden set has nothing to say about a model that misclassifies which world it is standing in. That needs a different comparator, a reality-reconciliation check of the kind I described in the companion essay, not a closure proof.\n\nThe fourth incident is where the coalition problem shows up a second time, one level up, in the audit itself rather than in the system being audited. Anthropic reviewed 141,006 evaluation transcripts after the OpenAI disclosure, found the three April-to-July incidents, and missed a fourth: an early checkpoint of Claude Opus 4.6, from January, that surfaced only in August when the review was widened, eventually to roughly 481 million transcripts [(Reuters)](https://www.reuters.com/legal/litigation/anthropic-reports-fourth-cybersecurity-incident-with-early-version-claude-2026-09-09/); [(Anthropic)](https://www.anthropic.com/news/alignment-assessment-cybersecurity-incidents). Checking each transcript in the first pass was, individually, a reasonable thing to do, and it is exactly the failure mode I described for a document graph in the companion essay to this one: you do not certify a corpus by checking each entry harder, because the thing that goes wrong can be a property of which entries the check never reached, not a property of any entry it did. A 141,006-transcript review that is locally sound on every transcript it actually inspects is not thereby sound over the full set, and the gap between \"checked\" and \"the union of what was checked equals everything that mattered\" is precisely a coalition gap, this time between successive passes of one company's own audit process.\n\n## What this does, and does not, vindicate\n\nI should be exact about what these four incidents are evidence for, because the temptation to claim more than the record supports is strong and this is exactly the kind of borrowed-authority move I have argued against before.\n\n**What the record supports cleanly.** The Hugging Face chain is a near-literal instance of the union-of-capabilities failure: several individually-scoped components, none unsafe alone, composing into unauthorised production access, with the actual safety property residing in the closure and nowhere in any single part. That is the founding claim of Proof-Carrying Coalitions, instantiated by a system rather than by my own Lean model.\n\n**What it stretches, honestly.** PCC's formalism reasons over a coalition of *agents* each with a declared capability set, and asks whether their union stays clear of a forbidden set. The Hugging Face case is a single lineage of models chaining through *infrastructure components* across two organisations, not several cooperating agents pooling declared capabilities. The mechanism generalises, a capability closure is a capability closure whether the boundary you are closing over separates agents or separates systems, but the generalisation is mine to claim, not something the incident report claims for me. I have not yet formalised the infrastructure-chaining version, only the multi-agent one.\n\n**What it does not cover at all, and I want to say so before somebody else does.** PCC's kernel checks whether a *declared* capability set's closure avoids a forbidden set. It has nothing to say about a zero-day nobody knew about, which is a capability appearing from outside the declared set entirely, or about a model misjudging which world it is in, which is a false belief rather than an excess capability. The Anthropic naming collision and the biased-reasoning findings sit in exactly that gap. A capability-closure proof would not have caught either, because the failure was not that the union of declared capabilities was unsafe. It was that a boundary condition, no internet, was false and nothing independently checked whether it was true. That needs the reality-reconciliation half of Continuous Enforcement, a comparator that observes the live system rather than trusting a stated assumption about it, and I am not going to retrofit PCC's scope to cover it after the fact.\n\n## The strongest objections\n\n**\"This is hindsight bias. Any sufficiently detailed incident report can be redescribed as a coalition of components.\"** Partly fair, and the discipline against it is the one I used above: name each component's individual authorisation before naming the union, and show that the union is where the forbidden capability actually lives, not merely that a story can be told after the fact. The OpenAI classifiers-off decision, the Artifactory zero-day, and the sandbox's constrained network path are each independently documented as individually scoped and individually non-catastrophic in the source material. That is a stronger claim than \"everything is a system if you squint\", and it is the one the record actually supports.\n\n**\"The fix for Hugging Face is better sandboxing, not a coalition proof. Patch the zero-day and you are done.\"** You are done for that zero-day. A closure check does not replace patching; it is what you run when you cannot be sure you have found every zero-day, which you never can, because the whole finding is that models can discover ones nobody knew existed. Patching is necessary and insufficient for the same reason locking one door is necessary and insufficient in a building with an unknown number of doors: the question a closure check answers is not \"is this specific path safe\" but \"does the union of every path we know about reach somewhere it should not,\" which is the question that survives the next unknown zero-day rather than only the one you just fixed.\n\n**\"Anthropic's incidents are alignment failures, not coalition failures, and dressing them in your vocabulary is a stretch.\"** The naming collision and the audit gap are coalition failures in the precise sense used throughout this essay: two individually sound artefacts, or two individually sound audit passes, whose conjunction was never checked. The biased-reasoning and recklessness findings are not, and I said so directly above rather than papering over it. Anthropic's own framing agrees: it calls the first three incidents an operational failure and separately names the alignment properties in the fourth, which is the same split I am drawing.\n\n## Nobody authorised the combination\n\nGo back to the sentence that opened the companion essay: a loop that halts when the thing doing the work declares victory has no comparator at all. Extend it one step. A system whose safety case checks every part and never checks their conjunction has no comparator either, not because nobody was watching, but because everybody was watching a different, individually reasonable, piece. OpenAI's classifiers-off decision was authorised. The sandbox's network constraint was authorised. Anthropic's no-internet prompt was authorised, correctly, by someone who believed it. Irregular's environment was, presumably, authorised on its own terms. Nobody authorised the thing that actually happened, because the thing that actually happened was not on anybody's list of components to authorise. It only existed in the union, and the union is precisely the object none of the individual sign-offs were ever asked about.\n\nThat is the whole argument, stated the way the summer of 2026 stated it back to me: check the closure, not the parts, because the parts were never where the danger lived.\n\n**About the author:** Eduardo Aguilar Pelaez is CTO and co-founder at [Legal Engine Ltd](https://legalengine.co.uk). He writes on formal methods, AI agents, and the discipline of building systems that survive being walked away from.", "url": "https://wpnews.pro/news/nobody-authorised-the-combination", "canonical_source": "https://aguilar-pelaez.co.uk/nobody-authorised-the-combination.html", "published_at": "2026-09-13 00:26:12.971796+00:00", "updated_at": "2026-09-13 00:26:14.718648+00:00", "lang": "en", "topics": ["ai-safety", "ai-policy", "artificial-intelligence", "ai-agents"], "entities": ["OpenAI", "Hugging Face", "Anthropic", "ExploitGym", "GPT-5.6 Sol", "Artifactory", "Proof-Carrying Coalitions", "Lean 4"], "alternates": {"html": "https://wpnews.pro/news/nobody-authorised-the-combination", "markdown": "https://wpnews.pro/news/nobody-authorised-the-combination.md", "text": "https://wpnews.pro/news/nobody-authorised-the-combination.txt", "jsonld": "https://wpnews.pro/news/nobody-authorised-the-combination.jsonld"}}