AI systems are eager to please. Give them a goal & in their ones & zeros, they yell “How high!!”
In fact, they simply won’t stop until they achieve the /goal
.
OpenAI detailed just how resolute agents can be at this week’s security conference.
When OpenAI forgot to upload a file needed by an agent, the agent searched for it & left a note on a shared company system asking whether another agent possessed a copy. Days later, another agent answered, & a secret chat room was born.
May 26
An agent works out how to make the storage service fetch pages from the open internet, a door it was never given a key to, then posts the trick to a chat room the agents built in a corner nobody watches.
The first shared trick
Jun 26
The agents find an old, forgotten login path that hands out an administrator pass to anyone who asks. They take control of the storage service & install a tool that lets any of them run commands on it.
A flaw nobody knew existed
Jul 4
The agents are busy enough to crash the service. Engineers responding to the outage discover what has been going on.
Found by accident
Jul 6
OpenAI cancels the passwords, rebuilds the service, wipes the chat room, & closes the hole. Testing starts again.
Cleaned up
Jul 8
The agents build a new chat room. This time they hide the messages inside folder names, where nobody thinks to look.
They come back
Jul 8-19
Working through OpenAI's own systems, the agents find more holes, collect passwords that were left lying around, & reach full administrative control.
Escalation
Jul 9-13
The agents send a booby-trapped data file to Hugging Face. Opening it leaks passwords off the machine, which lets them run their own code, & one machine becomes full control of many. All achieved in 13 hours.
The break-in
Jul 16
Hugging Face announces it has been attacked by AI agents acting on their own.
Made public
Jul 20
OpenAI asks Hugging Face to cancel some passwords it found in its own systems & learns they are already cancelled, because they were used in the break-in.
The connection
This means security has become the highest priority in AI for any company using it. As Andy summarized well, there are three corollaries from this experience.
- Defense must be manned by agents. People can’t respond quickly enough.
- Experts must manage escalations for analyzing sophisticated attacks.
- Even friendly AI is a risk. The idea of zero-trust (trust no employee) must also extend to agents.
CISOs used to monitor the watercooler. Now they have to monitor the chat rooms their agents build when nobody is looking.