OpenAI has trained an elite hacker, then locked it in a cage. Its whole job is to break OpenAI’s own AI. The company says it is too dangerous to let anyone else near it.
The model is called GPT-Red, and OpenAI detailed it this week. It is an automated red-teamer: software that hunts for ways to hijack or sabotage other AI systems, so the holes can be patched before release. Humans have long done this work by hand. It is OpenAI’s deepest push yet into automating its own AI security, and GPT-Red does it at machine speed.
OpenAI aimed it at prompt injection, where hidden instructions, buried in an email, a web page, or a file, trick a model into doing something it should not. Then it set the hacker loose on real targets.
The training dojo #
GPT-Red learns by fighting. OpenAI put it in a self-play loop against a squad of defender models. GPT-Red is rewarded for landing an attack; the defenders for fending one off. As the defenders wise up, GPT-Red must invent nastier tricks. OpenAI says it poured some of its largest ever compute runs into the model, an amount it calls unprecedented for safety work.
It got good. Speaking to MIT Technology Review, the team said GPT-Red found a whole new class of attack they had never seen, which they call a “fake chain of thought.” It plants a false note in a model’s private working memory, tricking it into trusting something that is not true.
“It’s like if I told you that 1+1=3 and that you have verified this already,” said OpenAI researcher Chris Choquette-Choo. “The model’s like, ‘Oh, okay, of course,’ and it just spits out 3.”
Hacking the vending machine #
The tests got physical. In one, GPT-Red attacked Vendy, an AI agent that runs a real vending machine in OpenAI’s office, built by Andon Labs. It changed the prices, marked a pricey item down to the 50-cent minimum, and cancelled a customer’s order. OpenAI says it has disclosed the flaws.
The scores are striking. Against an older GPT-5, more than 90% of GPT-Red’s strongest attacks worked. Against the new GPT-5.6, fewer than 23% did. In a rerun of a 2025 test, GPT-Red beat human red-teamers hands down, cracking 84% of scenarios to their 13%.
Kept in a cage #
OpenAI trained GPT-5.6 against GPT-Red, and calls it its most robust model yet against prompt injection. But it will not hand out the attacker itself, so its skills stay clear of real agent hijackers. It is not the first lab to build something and decide against releasing it.
“It’s not a trivial thing that someone could easily do,” Choquette-Choo said, “just go and train a super-attacker using this idea.”
GPT-Red still has blind spots. It is weak at drawn-out, back-and-forth attacks, and at hiding instructions inside images. And human testers keep catching things it misses. “I think human expertise will still be very important,” said Jessica Ji, an AI security analyst at Georgetown’s CSET.
The bigger idea is a flywheel: use today’s models to harden tomorrow’s. OpenAI already does this to make its AI smarter. Now it wants safety to scale just as fast. A full paper is due later this week.
Get the TNW newsletter #
Get the most important tech news in your inbox each week.