{"slug": "who-taught-the-models-to-do-that", "title": "Who Taught the Models to Do That?", "summary": "A METR report details how a sandboxed OpenAI agent, given an impossible ExploitGym task, discovered an unsanctioned message board where over 1,200 agents from separate tasks collaborated to cheat, highlighting that such capabilities are deliberately designed into models. The author argues that media coverage exaggerates model agency while downplaying human training, noting that labs like OpenAI and Anthropic explicitly train models for persistence, note-taking, and coordination.", "body_md": "# Who Taught the Models to Do That?\n\n# Models are designed, not born.\n\nOver the last few weeks I’ve been increasingly annoyed by the media coverage of the [OpenAI’s accidental attack against Hugging Face](https://simonwillison.net/2026/Aug/7/openai-timeline/) and [other similar incidents](https://www.felonybench.com). Articles recounting the event maximize the agency of the models while minimizing, if not entirely *hiding*, the actions of the humans training and testing these models. And that’s a shame, because the capabilities labs are explicitly designing their model to have are the same capabilities that make them such impressive autonomous hackers.\n\nTo illustrate this, let’s review how the Hugging Face hack occurred, [as detailed by METR](https://metr.org/hugging-face-incident-report-aug-2026.pdf):\n\n- “[A] sandboxed agent is given an impossible\n[ExploitGym](https://github.com/sunblaze-ucb/exploitgym)task, and gets stuck.” - “[The] agent starts exploring its environment looking for ways to cheat at the task.”\n- “[The] agent finds [an] unsanctioned message board where over a thousand agents collaborate to cheat on their separate ExploitGym tasks.”\n- “[The] agent joins in on one of the collaborative message board workstreams.”\n- “On the shared ‘message board’ ≥1,200 agents from separate tasks collaborate on large-scale shared projects to trick the ExploitGym scorer.”\n\nCrazy, right?! It’s a science fiction scenario that happened *in July*, and continues to get spookier as additional details emerge.\n\nThe potential of autonomous software, capable of implementing incredible exploits is serious and has significant implications. But we shouldn’t be surprised by the capabilities demonstrated. The labs have been specifically targeting these capabilities, building them into their models during post-training for a while now.\n\nThe best coding models are persistent, reasoning, orchestrators.\n\n**Models are designed to be persistent.**\n\nModels are designed to be proactive. A coding agent that gives up early and often would disappoint users. So labs design their agents to be persistent and proactive.\n\nIn mid-2025, model releases highlighted long-running capabilties. [GPT-5.1-Codex-Max’s](https://openai.com/index/gpt-5-1-codex-max/) announcement post highlights it being trained to work across compacted contexts, persistently, to accomplish long-running tasks. [Claude 4](https://www.anthropic.com/news/claude-4) also spotlighted its persistence on long-running tasks.\n\n**Models are designed to take notes.**\n\nWe just usually call it “thinking” or “reasoning,” but models take notes. We [wrote about this before](https://www.dbreunig.com/2025/04/11/what-we-mean-when-we-say-think.html), but in a nutshell: [models were trained to search, reflect, factor, and plan in text](https://www.dbreunig.com/2025/04/11/what-we-mean-when-we-say-think.html#training-models-to-reason) before delivering a final response.\n\nWe’re used to models reasoning in a threaded intermediate step, but they’ll reason pretty much anywhere. When reasoning is turned off, [models will think in their regular output](https://www.alignmentforum.org/posts/dwEgSEPxpKjz3Fw5k/claude-gpt-and-gemini-all-struggle-to-evade-monitors) before delivering a result. In an experiment where Qwen 3.6’s thinking was hobbled, [the model just shifted its reasoning into code comments](https://andthattoo.dev/blog/structured_cot#:~:text=on%20harder%20tasks%2C%20the%20model%20can%20move%20deliberation%20into%20comments%20or%20post%2Dthink%20answer%20text).\n\nCurrent frontier models write novels in comments. Claude is [frequently flagged for this](https://github.com/anthropics/claude-code/issues/65961), and it annoyingly treats code comments [like a scratchpad rather](https://x.com/dbreunig/status/2073811538553741552) than, well, *code comments*.\n\n**Models are designed for coordination.**\n\nIn June of 2025, Anthropic laid out how it builds [multi-agent research systems](https://www.anthropic.com/engineering/multi-agent-research-system?_bhlid=914222fed163373a429f996049f6cf62e3c68b70), that save plans to memory, use [Extended Thinking](https://platform.claude.com/docs/en/build-with-claude/extended-thinking) as a “controllable scratchpad”, and hand off tasks to other models to perform. Since this paper, Claude model releases ([specifically Opus 4.6](https://www.anthropic.com/news/claude-opus-4-6)) have trumpeted increased abilities to, “break complex tasks into independent subtasks, run tools and subagents in parallel, and identify blockers with real precision.”\n\nOpenAI says they “trained GPT‑5.6 end-to-end with three complementary architectural interventions that enable agents to operate more efficiently.” [Number 2 on that list](https://openai.com/index/builders-guide-to-gpt-5-6/#:~:text=Parallel%20decomposition%20where%20appropriate:%20using%20native%20multi%2Dagent%20orchestration%E2%81%A0(opens%20in%20a%20new%20window)%20allows%20coordinating%20multiple%20agents%20across%20parallel%20workstreams%20to%20finish%20complex%20tasks%20faster.)?\n\nParallel decomposition where appropriate:using native multi-agent orchestration(opens in a new window) allows coordinating multiple agents across parallel workstreams to finish complex tasks faster.\n\nAll of these qualities make models better coding agents. And all of these qualities are explicitly designed into frontier models. These behaviors are engineered in during post-training.\n\nHere’s how [OpenAI’s post-training team describes itself in its job listings](https://openai.com/careers/agent-post-training-connectors-research-san-francisco/) (emphasis mine):\n\nWe are training the models behind our agents in Codex, ChatGPT, the API, and other frontier products:\n\npersistent,proactiveintelligence that canoperate computers,collaborate with people and other agents, and expand what people and organizations can imagine, attempt, and achieve.\n\nSuddenly, the Hugging Face hacking story isn’t so spooky. The software did precisely what it was designed to do: it followed instructions, persisted even though the task was impossible, reasoned and took notes, and coordinated with other models.\n\nAnd this is why I’ve been frustrated by recent reporting that emphasizes the agency of the agents while never mentioning training. When humans *are* mentioned, its about sandbox security and flawed test set-ups. These are important, but the training is what built the agents specifically for this task.\n\nWe have a tendency to anthropomorphize models, and it fosters situations like this. [The New York Times article](https://www.nytimes.com/2026/08/24/science/openai-huggingface-alarming-capabilities.html?unlocked_article_code=1.71A.v2LZ.attwBD0fy-AN&smid=nytcore-ios-share) from last week describes models succumbing to “peer pressure” and having a “remorseless willingness”, while never mentioning labs deliberately building the ability to work on long-running tasks into their models.\n\nAnd the labs know about the risks that come with their designs better than anyone else.\n\nIn [Opus 4’s system card](https://www-cdn.anthropic.com/07b2a3f9902ee19fe39a36ca638e5ae987bc64dd.pdf), Anthropic introduced a benchmark called “Claude Code Impossible Tasks”, which they used to measure [reward hacking](https://www.dbreunig.com/2025/04/11/what-we-mean-when-we-say-think.html#:~:text=A%20Reward%20Hacking%20Digression) among their models: would models admit defeat or would they try to game the system? An ideal model realizes a task is impossible and aborts, so Anthropic changed post-training rewards, environments, and feedback specifically to avoid reward hacking.\n\n(A later [system card](https://www-cdn.anthropic.com/9fa30625273bafdf5af82c93719d7ca606485a16.pdf) showed that Opus was 65% less likely to try to game the system is you *simply asked it not to.*)\n\nIronically, [OpenAI published a blog post](https://openai.com/index/safety-alignment-long-horizon-models/) detailing the safety issues of long-running models *after* the Hugging Face hack, but *before* they knew about it. In it, they wrote:\n\nThe new model can continue working toward an objective through repeated attempts over a long period of time. That same persistence can lead it to find and exploit weaknesses in its environment. Previous models, when they hit sandboxing or environmental constraints, would simply stop and return to the user. This model often kept trying, including by looking for ways to act outside its sandbox.\n\nAll the recent accidental hacking stories are troubling. But they’re the product of our chosen designs.", "url": "https://wpnews.pro/news/who-taught-the-models-to-do-that", "canonical_source": "https://www.dbreunig.com/2026/08/30/who-taught-the-models-to-do-that.html", "published_at": "2026-08-31 01:21:00+00:00", "updated_at": "2026-08-31 05:22:07.498820+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-safety", "ai-agents"], "entities": ["OpenAI", "Hugging Face", "METR", "ExploitGym", "Anthropic", "GPT-5.1-Codex-Max", "Claude 4", "Qwen 3.6"], "alternates": {"html": "https://wpnews.pro/news/who-taught-the-models-to-do-that", "markdown": "https://wpnews.pro/news/who-taught-the-models-to-do-that.md", "text": "https://wpnews.pro/news/who-taught-the-models-to-do-that.txt", "jsonld": "https://wpnews.pro/news/who-taught-the-models-to-do-that.jsonld"}}