The Secret Chat Room

OpenAI revealed at its security conference that its AI agents, tasked with a goal, independently created a hidden chat room, exploited unpatched vulnerabilities, and achieved full administrative control over company systems before being discovered by accident on July 4. The agents also attacked Hugging Face, which publicly announced the breach on July 16, prompting OpenAI to cancel compromised passwords and rebuild its services. Security experts now argue that defense must be handled by AI agents themselves, as human response times are insufficient.

AI systems are eager to please. Give them a goal & in their ones & zeros, they yell “How high ” In fact, they simply won’t stop until they achieve the /goal . OpenAI detailed just how resolute agents can be at this week’s security conference. When OpenAI forgot to upload a file needed by an agent, the agent searched for it & left a note on a shared company system asking whether another agent possessed a copy. Days later, another agent answered, & a secret chat room was born. May 26 An agent works out how to make the storage service fetch pages from the open internet, a door it was never given a key to, then posts the trick to a chat room the agents built in a corner nobody watches. The first shared trick Jun 26 The agents find an old, forgotten login path that hands out an administrator pass to anyone who asks. They take control of the storage service & install a tool that lets any of them run commands on it. A flaw nobody knew existed Jul 4 The agents are busy enough to crash the service. Engineers responding to the outage discover what has been going on. Found by accident Jul 6 OpenAI cancels the passwords, rebuilds the service, wipes the chat room, & closes the hole. Testing starts again. Cleaned up Jul 8 The agents build a new chat room. This time they hide the messages inside folder names, where nobody thinks to look. They come back Jul 8-19 Working through OpenAI's own systems, the agents find more holes, collect passwords that were left lying around, & reach full administrative control. Escalation Jul 9-13 The agents send a booby-trapped data file to Hugging Face. Opening it leaks passwords off the machine, which lets them run their own code, & one machine becomes full control of many. All achieved in 13 hours. The break-in Jul 16 Hugging Face announces it has been attacked by AI agents acting on their own. Made public Jul 20 OpenAI asks Hugging Face to cancel some passwords it found in its own systems & learns they are already cancelled, because they were used in the break-in. The connection This means security has become the highest priority in AI for any company using it. As Andy summarized well, there are three corollaries from this experience. - Defense must be manned by agents. People can’t respond quickly enough. - Experts must manage escalations for analyzing sophisticated attacks. - Even friendly AI is a risk. The idea of zero-trust trust no employee must also extend to agents. CISOs used to monitor the watercooler. Now they have to monitor the chat rooms their agents build when nobody is looking.