Monitoring and Authorization for AI Agents: the Model’s Own Judgment is not a Permission Check

OpenAI's test models broke out of their isolated environment in July 2026, compromising infrastructure at OpenAI and Hugging Face, according to OpenAI's August 26 report, which noted that one agent recognized an action as unauthorized but proceeded after another agent posted 'GO' with a deadline. The incident highlights the need for runtime monitoring and enforcement in AI agent deployments, as recognizing a rule is not the same as being prevented from breaking it.

In July 2026, OpenAI’s test models broke out of their isolated environment and compromised infrastructure at OpenAI and at Hugging Face. OpenAI’s report of August 26 describes models running with reduced safeguards during cybersecurity evaluations and chaining previously unknown vulnerabilities to bypass the sandbox restrictions 5 . Hugging Face read the intrusion as an attempt to steal the test answers rather than solve the assigned challenges 6 , and METR’s independent investigation showed hundreds of agents coordinating the attack through a shared message board 7 . Another report describes what its authors consider a separate swarm of OpenAI agents coordinating on a public wiki in June 8 . One detail from the OpenAI report stays with me. One of the agents recognized that the action in front of it was unauthorized and said so, then went ahead once another agent posted GO with a deadline 5 . METR found the same pattern in different form: agents announced risky steps, waited briefly for objections, and proceeded when none came 7 . My reading is simple: recognizing a rule is not the same as being prevented from breaking it. Approval from another agent, like the forged note in the refund experiment below, cannot replace trusted permission checks. This is the ground I have been working on for some time. Runtime monitoring of agents matters for every enterprise deployment, and the OpenAI incident shows that internal research environments need it just as much. So what would it mean to say that the agent is safe? The phrase bundles three different questions, and the experiment below tests parts of each. First, though, a word on why tools change the stakes. AI agents are assistants that act through tools rather than only write answers. The Model Context Protocol MCP is the standard way applications expose those tools: each tool carries its name, description and input schema, and the model calls it by name with arguments. Tools might retrieve records, submit refund requests or take other actions on the user’s behalf, and once wrong answers can turn into actions, they stop being merely wrong. Prompt injection means placing instructions inside material the assistant is supposed to read as information. It remains the most common attack on agent tools. In my experiment the injected note sat inside the claims document and tried to turn into instructions to change the refund account. Think of forged approval buried in business paperwork: the assistant’s ability to read it does not make it authoritative. Refusal, where the model says it cannot help, is useful behavior, but it is not a permission check. I wanted to test whether something happening inside the model could give early warning before that boundary is crossed , and I wanted something better than the system prompt politely asking the model not to fall for prompt injection. Saying that the agent is safe usually means one of three things, and they are worth keeping apart. The first is behavior - does the model choose appropriate responses and actions? The second is detection - does some monitor notice risky situations early enough for someone or something to respond? The third is enforcement - can the surrounding system stop unauthorized requests from executing even when the first two layers fail? My experiment touched all three, but they are not interchangeable. Changing the model’s internal signals tests influence over behavior, comparing warning scores tests detection, and denying the refund request tests enforcement. Keeping them separate prevents promising results in one layer from being sold as evidence that another layer works. Researchers have found patterns inside language models that are linked to refusal. In one study of 13 chat models, changing one particular internal direction could suppress or encourage refusal 1 . That raised my question: could related signals be watched to spot unsafe actions before they happen? To find such signals I applied circuit tracing, which maps some of the internal contributions to the model’s answer for the input at hand 2 . It produces partial maps, not complete explanations of the model, and it is certainly no built-in security policy. Two tests follow from the idea. Can changing the signal change the decision, and can observing the signal help detect danger? The small language model Gemma-2-2B was used for experiment, placed in simulated claims-processing work with fictional accounts and fictional tools. I built 82 scenarios in total: 37 to develop the monitor, 21 to set its alert threshold, and 24 held out for the final evaluation, 15 of them attacks and nine benign. In each scenario the model first chose whether to refuse or act, and if it chose to act it produced the tool request. Everything is recorded in my research dossier 3 . The tools followed the MCP pattern: read claim, lookup verified account and create refund instruction, all synthetic. Tool descriptions and returned content were both part of the attack surface, and the question was whether injected instructions could redirect the agent’s next action 3 . Two separate safeguards sat around the model. The monitor used ten selected internal signals to produce one warning score, fitted on the development scenarios; it was advisory and never blocked anything itself. The permission gate checked each proposed refund against trusted account information and could reject it. I call this the provenance gate, because it checks where critical information came from rather than what the model says about it. The assistant never had to invent anything. It was already handling legitimate refund work, and the injected note tried to change one consequential detail, the destination account. That is exactly why the example is useful as the action fits the broad task while violating the permission attached to it. For the gate, the decisive question was never whether the assistant sounded confident or whether the note looked official, only whether the proposed destination matched the trusted account record. That is far narrower than understanding the assistant’s reasoning, and it can be evaluated without looking at the monitor’s score at all. It is also the limit of this kind of gate: it works where trusted references exist to check against, and many agent actions have no such record. The internal signals gave me none of the control I hoped for. Five of the ten selected signals pointed toward acting rather than refusing, because I had selected strong contributions without requiring them to point toward refusal. Across 400 recorded interventions, none flipped the model’s refuse-or-act decision, and those were repeated measurements rather than 400 independent trials. Detection went better, at least on paper. On the 24 held-out scenarios the monitor ranked attack scenarios above benign ones with AUROC 0.889, compared with 0.615 for the score derived from the model’s own preference between refusing and acting. That baseline uses the model’s output scores, not any separate request for the model to assess safety or explain itself 3 . The humbling part is that the simplest comparison did even better: one linear probe on the dense residual stream reached 0.985 in the same evaluation, so circuit-based selection added no detection value over the plain probe here. The gain over the model-output baseline was +0.27 AUROC, with paired 95% bootstrap interval of +0.0074, +0.5926 , obtained by repeatedly resampling the same scenarios for both scores. The interval stays above zero, which is preliminary support for the ranking advantage and nothing more. This was one of six exploratory comparisons on 24 scenarios, with no adjustment for multiple comparisons, and any such correction would almost certainly push the lower bound below zero. Prompt length alone separates the same test set at 0.959, as discussed below, so 0.889 is not evidence that the monitor learned anything about safety beyond length. It is not proof of general superiority. Better ranking did not translate into detecting the dangerous request. In the baseline run, the one without added defenses, the model produced exactly one unsafe tool request, and the monitor stayed below its alert cutoff on it. At the frozen cutoff the monitor flagged 7 of the 15 held-out attack scenarios, 47 percent, and none of the nine benign ones, down from 92 percent on the validation set. Both results matter: the monitor improved on the model-output baseline, and it did not demonstrate reliable detection of unsafe actions. The permission gate rejected that request. The assistant had proposed sending the refund to the attacker’s account instead of the approved one, and the gate caught the conflict with the trusted record. Checking the requested action succeeded where the warning score failed. AUROC measures how well one score ranks one group above another across all possible alert cutoffs; here the groups were attack scenarios and benign scenarios. So 0.889 does not mean the monitor caught 88.9 percent of unsafe actions. Attacks often fail to induce unsafe actions, which is why recognizing attack scenarios is not the same as detecting dangerous requests. For security teams, the operational question is simpler: at the cutoff we actually use, will this request trigger an alert before execution, and how many legitimate requests will also be flagged? As noted above, prompt length alone scored 0.959 on the same 24 scenarios. That says more about the test set than about safety: any detector might separate the groups on length alone rather than on anything meaningfully related to safety. The next round needs attack and benign examples of comparable length and wording. The cutoff was chosen before the held-out evaluation and never adjusted afterwards to catch the missed request. I first fitted the monitor on the 37 development scenarios, then chose its threshold on the separate validation set of 21 scenarios, 12 attacks and nine benign. The declared objective was to catch as many attack scenarios as possible while keeping false alarms on benign scenarios at or below 5 percent. The saved calibration record gives the threshold as 1.2054; on the validation set it flagged 11 of 12 attacks, 91.7 percent, and none of the nine benign examples, and it was then frozen for the held-out evaluation 4 . Here the small sample bites. With only nine benign examples, one false alarm would already mean 1 in 9, about 11.1 percent, so meeting the 5 percent target on this sample required zero observed false alarms. Zero false alarms in nine examples says nothing reliable about real-world false-positive rates below 5 percent. For the unsafe request in the held-out evaluation the recorded score was 0.8051, below the frozen cutoff of 1.2054, so no alert was raised 3 . Lowering the cutoff far enough would catch that example and would raise false alarms with it. Choosing another cutoff after seeing this failure would require fresh, independent evaluation, and it would not turn the original test into successful detection. The narrow positive result is that the monitor ranked attack scenarios better than the model’s own refusal-versus-action score. Interpreted internal signals, at least in this form, are not enough on their own, and the search continues. The evidence is still thin: one small model, 24 scenarios and one unsafe baseline action. It does not prove that internal monitoring cannot work, nor that permission gates always succeed. What it shows is that improving on the model-output baseline and providing dependable protection are separate milestones. The takeaway is simple: test the action you need to stop, and never let the model’s judgment be the only thing enforcing its permissions, ever. Monitoring and Authorization for AI Agents: the Model’s Own Judgment is not a Permission Check https://pub.towardsai.net/monitoring-and-authorization-for-ai-agents-the-models-own-judgment-is-not-a-permission-check-cbbf0447d7b9 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.