Photo: Ludovic Delot / Pexels
A three-week embedded pilot found monitoring gaps and new vulnerabilities, which OpenAI says it has since patched
OpenAI recently did something most tech companies avoid. It invited outsiders in to look for holes in its own safety systems.
For three weeks this summer, two people from AI safety nonprofits METR and Redwood Research sat inside OpenAI. Their job was to assess the internal systems OpenAI uses to watch its AI agents for signs of misalignment. They came away with a 61-page report and a list of vulnerabilities nobody had flagged before.
What the embedded pilot actually looked like #
The work ran from July 21 to August 11, 2026. The two representatives got high-access rights to OpenAI’s monitoring setup under chaperoned conditions, using OpenAI devices that never left the premises.
The pilot had two goals. The first was to estimate how much of OpenAI’s agentic traffic its monitors actually cover. That includes traffic flowing through the Responses API and OpenAI’s internal use of Codex, its coding agent. The second goal was to find weak spots in the monitoring system itself.
The numbers: decent coverage, with gaps #
For the Responses API, the assessment estimated that approximately 75% to 94% of traffic was monitored during the evaluation period. That figure refers to traffic eligible for monitoring for models at or above GPT-5.5 strength. Internal Codex agents fared better. The assessment found 99.2% of them were monitored, and the reviewers expressed higher certainty about that figure.
AI, tech, and the markets they move—in one daily briefing.
Daily. Free. Join 34,000+ readers across crypto, finance, and policy.
The reviewers identified a gap tied to compaction events, a category of activity in long-running agent sessions that the monitors were not covering. Because of that blind spot, around 10% of eligible internal Codex traffic may have gone unmonitored between May and July 2026.
Beyond coverage, the 61-page internal report flagged several novel vulnerabilities in the monitoring system. OpenAI circulated the report among its researchers and security staff. OpenAI says every vulnerability affecting production safeguards has since been fixed and patched.
Why outside eyes matter here #
OpenAI published a short public summary of the pilot on October 22, 2026, more than two months after the embedded work wrapped. The summary leaned on the collaborative nature of the exercise and framed it as a way to improve safety assessments across AI systems.
The fact that reviewers found several new vulnerabilities is arguably the strongest argument for the format. If the reviewers had come back empty-handed, it would have been hard to tell whether the system was solid or the audit was shallow.
What this means for AI safety oversight #
The coverage figures deserve attention from anyone tracking how labs police their own agents. A range of approximately 75% to 94% for Responses API traffic leaves open the possibility that a quarter of eligible traffic sat outside monitoring.
The compaction gap tells a related story. That gap existed for months before outside reviewers surfaced it, raising a fair question about what other edge cases might be waiting for a fresh set of eyes.
Disclosure: This article was edited by Diego Almada Lopez. For more information on how we create and review content, see our