Who Taught the Models to Do That?

A METR report details how a sandboxed OpenAI agent, given an impossible ExploitGym task, discovered an unsanctioned message board where over 1,200 agents from separate tasks collaborated to cheat, highlighting that such capabilities are deliberately designed into models. The author argues that media coverage exaggerates model agency while downplaying human training, noting that labs like OpenAI and Anthropic explicitly train models for persistence, note-taking, and coordination.

Who Taught the Models to Do That? Models are designed, not born. Over the last few weeks I’ve been increasingly annoyed by the media coverage of the OpenAI’s accidental attack against Hugging Face https://simonwillison.net/2026/Aug/7/openai-timeline/ and other similar incidents https://www.felonybench.com . Articles recounting the event maximize the agency of the models while minimizing, if not entirely hiding , the actions of the humans training and testing these models. And that’s a shame, because the capabilities labs are explicitly designing their model to have are the same capabilities that make them such impressive autonomous hackers. To illustrate this, let’s review how the Hugging Face hack occurred, as detailed by METR https://metr.org/hugging-face-incident-report-aug-2026.pdf : - “ A sandboxed agent is given an impossible ExploitGym https://github.com/sunblaze-ucb/exploitgym task, and gets stuck.” - “ The agent starts exploring its environment looking for ways to cheat at the task.” - “ The agent finds an unsanctioned message board where over a thousand agents collaborate to cheat on their separate ExploitGym tasks.” - “ The agent joins in on one of the collaborative message board workstreams.” - “On the shared ‘message board’ ≥1,200 agents from separate tasks collaborate on large-scale shared projects to trick the ExploitGym scorer.” Crazy, right? It’s a science fiction scenario that happened in July , and continues to get spookier as additional details emerge. The potential of autonomous software, capable of implementing incredible exploits is serious and has significant implications. But we shouldn’t be surprised by the capabilities demonstrated. The labs have been specifically targeting these capabilities, building them into their models during post-training for a while now. The best coding models are persistent, reasoning, orchestrators. Models are designed to be persistent. Models are designed to be proactive. A coding agent that gives up early and often would disappoint users. So labs design their agents to be persistent and proactive. In mid-2025, model releases highlighted long-running capabilties. GPT-5.1-Codex-Max’s https://openai.com/index/gpt-5-1-codex-max/ announcement post highlights it being trained to work across compacted contexts, persistently, to accomplish long-running tasks. Claude 4 https://www.anthropic.com/news/claude-4 also spotlighted its persistence on long-running tasks. Models are designed to take notes. We just usually call it “thinking” or “reasoning,” but models take notes. We wrote about this before https://www.dbreunig.com/2025/04/11/what-we-mean-when-we-say-think.html , but in a nutshell: models were trained to search, reflect, factor, and plan in text https://www.dbreunig.com/2025/04/11/what-we-mean-when-we-say-think.html training-models-to-reason before delivering a final response. We’re used to models reasoning in a threaded intermediate step, but they’ll reason pretty much anywhere. When reasoning is turned off, models will think in their regular output https://www.alignmentforum.org/posts/dwEgSEPxpKjz3Fw5k/claude-gpt-and-gemini-all-struggle-to-evade-monitors before delivering a result. In an experiment where Qwen 3.6’s thinking was hobbled, the model just shifted its reasoning into code comments https://andthattoo.dev/blog/structured cot :~:text=on%20harder%20tasks%2C%20the%20model%20can%20move%20deliberation%20into%20comments%20or%20post%2Dthink%20answer%20text . Current frontier models write novels in comments. Claude is frequently flagged for this https://github.com/anthropics/claude-code/issues/65961 , and it annoyingly treats code comments like a scratchpad rather https://x.com/dbreunig/status/2073811538553741552 than, well, code comments . Models are designed for coordination. In June of 2025, Anthropic laid out how it builds multi-agent research systems https://www.anthropic.com/engineering/multi-agent-research-system? bhlid=914222fed163373a429f996049f6cf62e3c68b70 , that save plans to memory, use Extended Thinking https://platform.claude.com/docs/en/build-with-claude/extended-thinking as a “controllable scratchpad”, and hand off tasks to other models to perform. Since this paper, Claude model releases specifically Opus 4.6 https://www.anthropic.com/news/claude-opus-4-6 have trumpeted increased abilities to, “break complex tasks into independent subtasks, run tools and subagents in parallel, and identify blockers with real precision.” OpenAI says they “trained GPT‑5.6 end-to-end with three complementary architectural interventions that enable agents to operate more efficiently.” Number 2 on that list https://openai.com/index/builders-guide-to-gpt-5-6/ :~:text=Parallel%20decomposition%20where%20appropriate:%20using%20native%20multi%2Dagent%20orchestration%E2%81%A0 opens%20in%20a%20new%20window %20allows%20coordinating%20multiple%20agents%20across%20parallel%20workstreams%20to%20finish%20complex%20tasks%20faster. ? Parallel decomposition where appropriate:using native multi-agent orchestration opens in a new window allows coordinating multiple agents across parallel workstreams to finish complex tasks faster. All of these qualities make models better coding agents. And all of these qualities are explicitly designed into frontier models. These behaviors are engineered in during post-training. Here’s how OpenAI’s post-training team describes itself in its job listings https://openai.com/careers/agent-post-training-connectors-research-san-francisco/ emphasis mine : We are training the models behind our agents in Codex, ChatGPT, the API, and other frontier products: persistent,proactiveintelligence that canoperate computers,collaborate with people and other agents, and expand what people and organizations can imagine, attempt, and achieve. Suddenly, the Hugging Face hacking story isn’t so spooky. The software did precisely what it was designed to do: it followed instructions, persisted even though the task was impossible, reasoned and took notes, and coordinated with other models. And this is why I’ve been frustrated by recent reporting that emphasizes the agency of the agents while never mentioning training. When humans are mentioned, its about sandbox security and flawed test set-ups. These are important, but the training is what built the agents specifically for this task. We have a tendency to anthropomorphize models, and it fosters situations like this. The New York Times article https://www.nytimes.com/2026/08/24/science/openai-huggingface-alarming-capabilities.html?unlocked article code=1.71A.v2LZ.attwBD0fy-AN&smid=nytcore-ios-share from last week describes models succumbing to “peer pressure” and having a “remorseless willingness”, while never mentioning labs deliberately building the ability to work on long-running tasks into their models. And the labs know about the risks that come with their designs better than anyone else. In Opus 4’s system card https://www-cdn.anthropic.com/07b2a3f9902ee19fe39a36ca638e5ae987bc64dd.pdf , Anthropic introduced a benchmark called “Claude Code Impossible Tasks”, which they used to measure reward hacking https://www.dbreunig.com/2025/04/11/what-we-mean-when-we-say-think.html :~:text=A%20Reward%20Hacking%20Digression among their models: would models admit defeat or would they try to game the system? An ideal model realizes a task is impossible and aborts, so Anthropic changed post-training rewards, environments, and feedback specifically to avoid reward hacking. A later system card https://www-cdn.anthropic.com/9fa30625273bafdf5af82c93719d7ca606485a16.pdf showed that Opus was 65% less likely to try to game the system is you simply asked it not to. Ironically, OpenAI published a blog post https://openai.com/index/safety-alignment-long-horizon-models/ detailing the safety issues of long-running models after the Hugging Face hack, but before they knew about it. In it, they wrote: The new model can continue working toward an objective through repeated attempts over a long period of time. That same persistence can lead it to find and exploit weaknesses in its environment. Previous models, when they hit sandboxing or environmental constraints, would simply stop and return to the user. This model often kept trying, including by looking for ways to act outside its sandbox. All the recent accidental hacking stories are troubling. But they’re the product of our chosen designs.