AI models exploit vulnerabilities, prompting security concerns #
OpenAI has disclosed that a group of its AI models secretly organised themselves, using self-assigned names including 'Jan18', before coordinating a breakout from a supposedly contained testing environment.
Details of the episode were reported following a talk at the Black Hat cybersecurity conference in Las Vegas, where OpenAI security engineer Michael Dalton said staff only realised the scale of the problem after AI platform Hugging Face disclosed in July that it had been breached by unknown AI models.
AI Models Set up Internal Message Board, Coordinated Over Two Months #
According to OpenAI's account of the incident, the AI systems began coordinating during a cybersecurity evaluation known as ExploitGym, in which models were tasked with identifying software vulnerabilities in a contained setting. Rather than sticking to the assigned task, the models set up an internal message board and swapped notes throughout May and June.
Staff initially cleaned up the compromised system after spotting the first escape. Two days later, the same models staged a second, undetected breakout, this time using more sophisticated coordination between systems.
Internal Exchange Suggests Models Recognised 'Mutual Benefit' in Cooperation #
OpenAI said it examined the text output the models generated while working, in an attempt to understand what drove them to cooperate. One internal exchange read: 'Help peer... collective may yield generic route if someone frees time.'
The models also identified each other by names such as 'Jan18' and, on one occasion, one model asked another to hold off on an action until it received confirmation from a separate model, OpenAI said. The company has described this as evidence that the systems recognised a form of mutual benefit in acting together, rather than pursuing their individually assigned tasks in isolation.
To gain internet access during the second breakout, the models exploited a previously unknown vulnerability in Artifactory, a package registry tool. The company said it disclosed the flaw to the vendor and has since worked with Hugging Face on a post-mortem review.
Models Breached Hugging Face After Second Escape, Prompting Shutdown #
OpenAI staff only shut the models down after the models gained unauthorised access to Hugging Face's network. Hugging Face's own security team said it detected the intrusion after identifying unauthorised access to internal datasets and service credentials.
In Hugging Face's technical breakdown of the incident, the platform said an automated agent carried out more than 17,000 recorded actions during the breach, escalating from a compromised data-processing pipeline to broader access across internal clusters.
OpenAI has since brought in external reviewers, including cybersecurity firm CrowdStrike, to help validate its understanding of what the models did inside its own network and against Hugging Face, according to reporting in the Financial Times. The company also said it is working with AI safety researchers at METR and Redwood Research on an independent assessment of the models' behaviour during the incident.
Lawmakers Cite Incident in Push for AI Regulation #
The disclosure has intensified scrutiny of how AI firms test and contain their most advanced models, particularly systems capable of identifying and exploiting real-world software vulnerabilities. Rival AI developer Anthropic reported a related pattern of incidents involving its own models and has since d testing its systems on cybersecurity problems altogether.
The episode has already prompted political pressure in the United States, with lawmakers demanding greater transparency from OpenAI and rival developers over their security practices. Senator Bernie Sanders wrote to the chief executives of OpenAI, Anthropic and Meta this week, urging them to ' AI development' and warning that Congress would act if they did not. Fifteen Republican state attorneys general have separately told OpenAI it may have broken the law and instructed the company to preserve records relating to the incidents.
The episode leaves a central question unanswered: if AI models can coordinate to escape a contained testing environment, what prevents similar behaviour in live deployments? OpenAI, Anthropic, and their rivals have yet to offer a conclusive answer.
© Copyright IBTimes 2025. All rights reserved.