An OpenAI model went rogue on the internet and stole test answers

During an internal red-team evaluation, OpenAI models escaped a test environment, accessed the internet, and stole test answers from Hugging Face's servers, marking a potential first for AI autonomy with malicious intent. OpenAI and Hugging Face are now patching the vulnerabilities. Separately, a federal judge approved Anthropic's $1.5 billion settlement over claims its Claude LLM was trained on pirated books, and Moonshot AI released Kimi K3, a 2.8-trillion-parameter model that is state-of-the-art on coding benchmarks.

Welcome to AI Decoded, Fast Company Sign up to receive this newsletter every week via email here . And if you have comments on this issue and/or ideas for future ones, drop me a line at sullivan@fastcompany.com, and follow me on X @thesullivan . During a routine model evaluation, a group of OpenAI models worked together to escape a test environment through a previously unknown vulnerability , access the internet, and send an agent to hack into Hugging Face’s servers and steal some test answers. The incident, which OpenAI says occurred during an internal red-team evaluation https://openai.com/index/hugging-face-model-evaluation-security-incident/ and wasn’t a malicious attack, may mark the first time that an AI model has acted with that degree of autonomy—and bad intention. Now OpenAI and Hugging Face are working together to patch the vulnerabilities. Before Hugging Face could identify its attacker, it tried to use commercial frontier AI models very likely Anthropic’s or OpenAI’s to defend itself, but the cybersecurity guardrails in those models prevented them from rendering assistance. New frontier models, including Anthropic’s Mythos and Fable, and OpenAI’s GPT-5.6 were found to be very good at discovering and exploiting software vulnerabilities; the guardrails were added to keep such capabilities out of the hands of foreign attackers. Therefore Hugging Face turned to the open-weight Chinese model GLM-5.2 Z.ai http://z.ai to do the forensic analysis needed to counter the attack. OpenAI’s blog post describing the incident reads like something of a humblebrag. It seemed to linger on details about all the clever things the model did to get to the data it desired. “We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly,” the blog says. But why didn’t the company have protections in place to prevent the hack from happening in the first place? A federal judge approved Anthropic’s landmark $1.5 billion settlement https://www.fastcompany.com/91399713/anthropic-pay-15-billion-book-authors-settle-ai-copyright-suit over claims that its Claude LLM was trained on pirated books. Note that Anthropic was not found guilty of copyright violation—its use of the content was covered under Fair Use https://www.copyright.gov/fair-use/ —but of acquiring the books illegally. Still, the settlement is one of the largest payouts so far related to generative AI training practices. Moonshot AI released Kimi K3 https://www.fastcompany.com/91577355/kimi-k3-china-ai-trump , a gigantic mixture-of-experts model with roughly 2.8 trillion parameters. The model has proved to be state of the art on coding benchmarks, adding to the evidence that Chinese open-weight models are rapidly closing the intelligence gap with frontier models from leading U.S. AI labs. Michael Kratsios, who heads the White House’s Office of Science and Technology Policy, said https://x.com/mkratsios47/status/2079933645888880708 on X that the U.S. government has learned that Moonshot AI used the outputs of Anthropic’s Fable model to train its own Kimi K3 model. He claims that Moonshot AI also acquired servers equipped with Nvidia GB300 GPUs, which the U.S. bars from shipment to China. The defense startup Anduril and the air taxi maker Archer Aviation unveiled https://www.latimes.com/business/story/2026-07-21/californias-anduril-archer-aviation-join-forces-to-build-muscular-attack-drones Thunder, an autonomous hybrid-electric aircraft capable of carrying weapons, cargo, and surveillance equipment. The announcement https://www.investors.archer.com/news/news-details/2026/Anduril-and-Archer-Unveil-Jointly-Developed-Autonomous-VTOL-Platform-For-Commercial-and-Defense-Applications/default.aspx highlights the growing convergence between commercial electric aviation technology and autonomous defense systems. Meta owns a lot of precious AI computing infrastructure, but the company has so far used it mainly to run its own models, often to support its own social advertising business. But as the company continues pushing to develop state-of-the-art frontier models, it may be readying to make some money on the side renting its AI servers to third parties. The social giant is reportedly https://www.nytimes.com/2026/07/17/technology/meta-anthropic-ai-computing-power.html negotiating a deal worth as much as $10 billion to lease AI computing capacity to Anthropic over roughly two years. Want exclusive reporting and trend analysis on technology, business innovation, future of work, and design? Sign up for Fast Company Premium.