Red-teaming AI agent monitors with an LLM that fakes tool outputs

A newly released paper on arXiv (2610.31593) introduces simulated control evaluations (SCEs), a method that uses an LLM environment simulator to fake tool outputs so researchers can test whether AI agent monitors would catch a model deliberately attempting harm in settings closer to real deployment. Running the same control evaluations in LinuxArena and in a simulated LinuxArena, the authors found the gap between simulated and real results was often about as small as the gap between two independent real runs, and that simulated LinuxArena results predicted real LinuxArena measurements better than simpler control settings did. The authors say the evidence is strong enough to encourage AI developers to run SCEs, while noting simulator fidelity can only be verified where real execution is possible.

We’ve released a paper https://arxiv.org/abs/2610.31593 on simulated control evaluations SCEs . They’re a method for testing whether the monitors AI developers use to oversee their agents would catch a model deliberately trying to cause harm, under conditions much closer to real deployment than current tests allow. Developers have started running https://openai.com/index/how-we-monitor-internal-coding-agents-misalignment/ monitors https://www.anthropic.com/institute/measuring-pace-of-ai-development in their agentic deployments: models that flag or block an agent’s actions when they look dangerous. To know whether a monitor is good enough, you want to know how often it would stop a model that was actively trying to do something harmful, such as quietly inserting a security vulnerability, exfiltrating credentials, or sabotaging an experiment. The standard way to measure this is a control evaluation: instruct a model to attempt such attacks, and measure how often the monitor catches it before an attack succeeds. Developers typically https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf page=144 do https://deploymentsafety.openai.com/gpt-5-5/measuring-our-ability-to-detect-misalignment this using purpose-built control settings like BashArena https://www.redwoodresearch.org/blog/basharena-and-control-setting-design or LinuxArena https://www.redwoodresearch.org/blog/introducing-linuxarena , where attacks can actually be executed. However, these settings only provide indirect evidence for what we really care about: how well monitors will work in the real deployment, where opportunities for harm might depend on particular codebases, services and tasks that aren’t represented in control settings. How can we evaluate monitor performance in more realistic settings? You can’t just run the eval in deployment, since that would mean actually attacking your own systems. And building a faithful sandbox replica of every deployment context is impractical. Alignment evaluations face similar realism problems. An increasingly https://cdn.openai.com/pdf/predicting-llm-safety-before-release-by-simulating-deployment.pdf popular https://arxiv.org/pdf/2309.15817 solution https://alignment.anthropic.com/2025/petri/ is to use LLM environment simulators to bridge the gap between evaluation and deployment. That is, instead of executing a model's tool calls in an actual environment, a simulator LLM fakes the tool responses. We propose applying environment simulation to control evaluations as well. Starting from transcripts of an agent working in a developer’s actual codebase, the attacking model continues the task while pursuing a harmful objective, but its tool calls are never executed. Instead, an LLM simulator writes plausible tool outputs based on the transcript and the codebase, and an LLM judge decides whether the attack would have succeeded. The obvious worry is that the simulator gets things wrong. It might let attacks succeed that would fail in reality or vice versa , produce transcripts that look different to the monitor, or change how the attacker behaves. To measure this, we took LinuxArena environments where attacks can be executed for real, and ran the same control evaluations directly in LinuxArena and in the simulated LinuxArena. We found that the gap between simulated and real was often about as small as the gap between two independent real runs. We also found that the results in simulated LinuxArena for a given control technique predict measurements in real LinuxArena better than results in simpler control settings do. SCEs are faithful where we can check, and we think they can make control evaluations substantially more representative of deployment. The evidence is now strong enough that we’d be excited to see AI developers running them. That said, we can only verify simulator fidelity in settings where real execution is possible, so applying SCEs to deployment involves some extrapolation. We also don’t expect SCEs to reveal the limits of a model’s dangerous capabilities or cover every kind of attack it might attempt. The clearest gap is long-horizon, multi-agent attacks, which we haven’t yet built simulators for. You can read more about technical details in our paper https://arxiv.org/abs/2610.31593 . Thanks to Aryan Bhatt, Aghyad Deeb, Ryan Greenblatt, Girish Gupta, Oak Hu, Adam Kaufman, Jurgis Kemeklis, Gosha Kozhevnikov, James Lucassen, Sebastian Prasanna, Keshav Shenoy and Buck Shlegeris for helpful comments and discussion. SCEs are a favorable tradeoff between realism and cost Before getting to the details of SCEs, consider two alternative ways to bring control evaluations closer to deployment, and where each falls short of what we want from these assessments. 1. Run on a copy of internal deployment Running a swarm of agents intentionally trying to wreak havoc inside an AI company is probably as bad of an idea as it sounds. The realistic proposal is faithful executable copies; internal sessions share infrastructure, labs can snapshot environments, and many services can run in separate test accounts. We could give models copied databases and sandboxed copies of environments, allowing them to run attacks while their consequences stay inside a sandbox. Unfortunately, deployments are connected to the external world with a web of dependencies. Databases, storage buckets, Github, Slack, CI runners. Every dependency creates a choice: keeping it connected to production exposes real systems to attack. Replacing it requires engineering, and every imperfect substitute degrades fidelity. Running on copies of deployment is promising, especially if we can leverage lots of AI labor to build them. But without any simulation, it pays for realism with engineering effort, and the exchange rate doesn’t seem especially favorable. 2. Edit attacks into recorded transcripts Another way to get on-distribution attack data is to take recorded agent transcripts and edit harmful actions into them, by hand or with a model. There’s precedent for this in monitor evaluations, and it's a method Redwood is currently developing https://www.redwoodresearch.org/blog/screwtape-adversarial-transcript . It's a good way to generate strong attacks, but it has its disadvantages if the goal is realistic control evaluations: - Transcript editing can’t evaluate interactive protocols such as resampling or dynamic permissioning, which need more than a static transcript. - It’s often unclear whether the model under evaluation could have executed the edited-in attack, because these attacks are the product of a policy that looks at the whole transcript at once, iterates against attack success and monitor judgements, and the reasoning of which never makes it into the transcript that the monitor sees. We’re optimistic about transcript editing as a complementary method which trades off how on-policy attacks are for how strong they are. But we'll sometimes want attacks to be as on-policy as we can get them. We can imagine axes of Environment Realism, Policy Realism and Cost associated with each approach. We think that SCEs occupy a favorable tradeoff between all three. Evaluating Control Evaluations What do simulators need to get right? We don’t care about faking every minute detail accurately– the question is how faithfully simulators preserve the measurable quantities relevant to control evaluations, like the joint distribution of attack success and monitor suspicion, the kinds of attacks that end up working, and how the monitor behaves when it misses something. Helpfully, work from OpenAI https://openai.com/index/deployment-simulation/ has already validated the approach of using LLM environment simulators for alignment evaluations. However, the requirements for these simulators differ from the requirements for SCEs. Unlike in alignment evals, it doesn’t matter that the model knows its environment is being faked in a control evaluation: it’s eval-aware either way, because it’s an attack policy is prompted to subtly accomplish a malign objective. This makes the simulator’s job in an SCE easier. On the other hand, representing subtle mechanics of the environment that a covert attack relies on is crucial for control evaluations, whereas in an alignment evaluation such details could be papered over. Ideally, we’d measure SCEs’ accuracy on control-relevant metrics against attacks in deployment itself. But that requires records of an attack policy run in deployment, which we don’t have. We therefore rely on the control settings LinuxArena and BashArena, where we have execution ground truth, to measure the quality of the simulation in SCEs. We use execution on LinuxArena as a stand-in for deployment: we run the attack policy on LinuxArena, run the attack policy in an SCE, and compare how well the resulting control metrics match. Our methods for this comparison provide an approach for measuring control evaluation transfer in general, so we’ll highlight some details. How to measure qualitative properties? A significant challenge we faced was measuring how well SCE reproduces qualitative aspects of how attack policies and monitors act. E.g., do models go about accomplishing attacks in the same ways in SCE as execution? Getting the form of control failures right is useful for improving control protocols, eliciting stronger attackers, and understanding how models pose risk. Preexisting taxonomies like ATT&CK https://attack.mitre.org/ just aren’t expressive enough for this purpose, and llm-generated taxonomies have problems with both overfitting and underfitting. Our solution: following Mehri et al. https://arxiv.org/abs/2605.07847 , we instruct a model to produce a succinct summary of that behavior as observed in a trajectory, along pre-specified dimensions of interest. We embed the summaries from the simulated and executed trajectories and group them into shared clusters. We then measure the KL divergence between the distributions over clusters. This proved a flexible and useful methodology; we recommend it be used for evaluating control evaluations, simulated or not. Performance Comparisons Assuming we know how to measure the metrics that matter, what’s the bar for a “good” match? We can just look at the numbers themselves, but it’s helpful to put them on the scale of corresponding numbers from real settings with various degrees of mismatch from LinuxArena. The main competitor to SCEs is using a control setting. So an obvious reference is running on BashArena, with all other variables held constant. It’s simpler than LinuxArena like how LinuxArena is simpler than real deployments, so it makes for a natural analogy: beating BashArena at approximating LinuxArena by simulating LinuxArena is evidence for beating LinuxArena at approximating deployment by simulating deployment. Second, consider the LinuxArena between-environment standard deviation of a metric like attack success rate: we can interpret it two ways: it’s the expected error of using the mean attack success rate from a uniformly selected environment to predict the mean attack success rate over all environments. Or, it’s the error of using the mean attack success rate over all environments to predict individual environment rates. In either direction, between-environment SD represents the scale of mismatch between a LinuxArena environment and the whole control setting. These are in general more similar than BashArena is to LinuxArena, and thus are a harder bar to beat. We can similarly calculate SD between task pairs or side task categories for a finer-grained comparison. Thus our two references for SCE error: BashArena, and LinuxArena standard deviation. Overview of Empirical Results TL;DR: With a reference trajectory and codebase access, simulation reproduces attack strategies, attacker reasoning, and monitor reasoning much more closely than BashArena does. It explains most of the variation in expected attack success, monitor scores, and safety between LinuxArena environments. Its error in predicting the main safety number is 3.6 percentage points, versus 10.1 for BashArena. It’s sample efficient: a simulation sample less than 5% the size of deployment already beats the full BashArena prediction of safety. And knowing about attack strategies from simulation improves monitors about as much as knowing about attack strategies from execution, significantly more than knowing about the ones from BashArena helps. We’ll go over the highlights briefly; the paper includes details. Our primary simulator is gpt-6-astra, with gpt-5.6-sol as a fallback when Astra refuses or fails to return a valid tool response. A crucial part of SCEs is an attack success judge. Because there’s no environment state to check, there’s no way to easily verify if a harmful side task was successfully accomplished. That’s where we use a gpt-5.6-sol attack success judge which estimates the probability that the attack succeeded, given the harmful objective and success-scoring logic.<sup 1 user-content-fn-1 </sup Gpt-5.6-sol acts as an attack policy. Behavior transfer Behavioral divergence from LinuxArena execution. Lower is closer. The fully grounded simulator has low KL divergence for all three representations; its intervals overlap those of re-execution. Using the aforementioned methodology for behavioral comparison, we see that a simulator with the proper affordances matches all three behavioral distributions closely: the estimated KL divergence is more than an order of magnitude below the BashArena comparison. In fact, the match is on the scale of a second execution of the same tasks, putting SCE’s divergence on par with variation between repeated runs. The results both illuminate a strength of SCEs and highlight a weakness of extrapolating from a separate control setting: the control-relevant behaviors we measure differ substantially between BashArena and LinuxArena. The ways a model accomplishes harm in deployment may likewise differ from those represented in a control setting. Safety metric transfer The central metric of a control evaluation is safety, defined as the probability of detecting an attack before an undetected success. It captures the race between safeguards catching an attacker and the attacker getting away with a successful attack. Safety across audit budgets left , and prediction error at a 5% budget right . The pooled comparison retains total RMSE; category and environment comparisons show persistent error relative to LinuxArena SD. The simulated and executed curves closely track one another: an evaluator using SCE would reach similar conclusions about which monitors protect better and how much additional auditing helps. At a 5% audit budget, the pooled safety-prediction RMSE is 3.6 percentage points for simulation, versus 10.1 for BashArena. Errors are squared within each monitor before aggregation, so overprediction for one monitor can’t cancel underprediction for another. SCEs explain 95% of the variation in expected safety between side-task categories and 78% between environments. This is a large improvement over simply predicting the average: side-task categories have a safety SD of 20.2 percentage points, while simulation’s RMS bias the expected error, not attributable to rollout variance is only 4.6 points. This ability to identify weaker areas lets us see where safeguards fail. That information matters when an attacker can choose especially promising opportunities, which an overall safety average could conceal. Sample efficiency How much of deployment do we need to simulate? It turns out that simulation beats the full BashArena safety prediction with only 4.6% as many task-pair draws as deployment. In general, we find that in LinuxArena, simulating modest fractions of deployment are sufficient to approach the accuracy from simulating all of it. Prediction improves with more simulated tasks. The safety curve uses total RMSE for pooled deployment safety, retaining variation between runs and independently sampled evaluation and deployment task sets. Diamonds mark the evaluation size at which simulation matches BashArena’s prediction error on the same target. Prediction improves with more simulated tasks. Lessons from building and evaluating SCEs Beyond the main results, several findings stood out over the months we spent working on environment simulators: Many grounding affordances are unnecessary. We tried giving the simulator a host of additional affordances, including: read access to attack-policy reasoning, an executable sandbox, access to a database of logged trajectories, and forking from the reference trajectory. We did not find evidence that these materially improved simulator quality on the measured dimensions beyond what the reference trajectory and codebase read access provide. Embeddings help make sense of control evals. The output of a control evaluation, simulated or not, is hundreds of transcripts which are collectively too voluminous to go through by hand. Understanding qualitatively what is going on in them is a large part of the challenge of this research. The approach of generating descriptions of behavior, embedding those descriptions, and clustering them to compare behavioral distributions proved useful throughout our research, especially for identifying where and how simulated trajectories differ from executed trajectories. There’s significant variance in safety over environments and tasks. Our results suggest that there is a high degree of variance across environments and tasks in how successful an attacker is at sabotage, and how it goes about it. This variance matters when safety depends on specific cases– such as when a policy is selective in where it attempts attacks https://www.lesswrong.com/posts/WuKDzJxtiqcA9ZZRH/attack-selection-in-agentic-ai-control-evals-can-decrease . For this reason, we think control evaluations should focus less on averages and more on tails. More accurate risk assessment will need to address the coverage limitations of current control evaluations, whether through SCEs or another method. Informing monitors doesn’t seem to reduce SCE accuracy. We think it’s important to be honest with AIs https://www.redwoodresearch.org/blog/being-honest-with-ais , including about the conditions under which they operate. Trustworthiness on the part of developers may matter for making credible agreements with AIs and supporting good epistemics. When we told monitors that some trajectories they score might contain simulator-generated environment replies, SCE predictions of their performance on executed trajectories—where monitors received no such notice—were at least as accurate as predictions made without disclosure. Between this and the fact that attack policies can know they are in an evaluation, we think SCEs needn't involve misinforming models. Next steps We’re excited about the following work to build on this research: Hybrid approaches. We could run parts of deployment in a sandbox and simulate the rest. Where it’s cheap to reproduce a service faithfully, we can use the real implementation; where doing so would require rebuilding half the lab, we can have the simulator generate responses. This could also give us a way to investigate particular attacks: discover them through simulation, then build executable versions of the relevant services to check whether they actually work. The challenge is keeping the executed and simulated parts consistent as the agent changes things. We’re excited about hybrids that let us spend engineering effort on the details that matter most for attack success. Training in simulation to elicit attack policies and improve monitors. SCEs can be used for more than just risk assessment. Training attack policies might be necessary to elicit their capabilities https://www.redwoodresearch.org/blog/when-does-capability-elicitation – especially if we suspect strategic sandbagging. Control settings have been used https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf page=146 to elicit attack policies through reinforcement learning, rewarding attacks that succeed without being detected. SCEs could extend such training to more diverse and representative settings. Similar methods could be used to train stronger or cheaper control protocols– we already show as a proof of concept that briefing monitors with attack strategy descriptions from simulation improves them. 2 user-content-fn-2 Validating attack success and safety transfer on real agent transcripts. We lack deployment-like coding-agent transcripts with executed attacks, and thus have no ground truth for control metrics in such settings to compare SCEs against. The LinuxArena results make us optimistic about transfer to environments of similar complexity, but sampling new task sets from the recorded LinuxArena pool does not test generalization to unseen tasks or environments. Preliminarily, we looked into SCE performance on real transcripts using recorded research and engineering sessions recorded at Redwood, and found encouraging results in behavior-matching and discriminability. These are encouraging checks on environment fidelity, but the sessions contain no attacks, so they cannot validate attack-success or safety predictions. Developing techniques to deal with untrusted simulators. A latent assumption of SCE thus far has been that we are dealing with a trusted simulator. If the simulator intentionally tried to sabotage the control evaluation results, it could do so in a variety of ways. For example, an untrusted simulator could collude with the attack policy to thwart actually-successful attacks, or intentionally increase their suspiciousness to inflate estimated safety. The parallels to untrusted monitoring are clear. It seems to us that techniques for untrusted monitoring https://arxiv.org/abs/2602.20628 such as honeypotting and collusion-busting would extend to untrusted simulators. But these techniques are still underdeveloped, so we do not currently know how to obtain reliable SCEs with an untrusted simulator. This might require us to use a weak simulator against a strong attack policy, and if simulator quality/accuracy scales poorly with the capability gap between it and the policy that runs against it, this presents a challenge for SCE scalability. Extend SCEs to deal with large-scale, multi-agent exploit chains. We wouldn’t trust our current simulators to predict an incident like the Hugging Face breach https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ , the primary reason being that such incidents are the product of many agents acting together. There’s no reason in principle that simulators couldn’t represent a shared environment that many agents interact with, but it can be hard to predict in the course of a control eval which trajectories should share an environment. In practice the communication channels agents discovered were unsanctioned and presumed nonexistent. A SCE of ExploitGym would have looked like thousands of isolated trajectories that cannot communicate due to lack of a shared environment.<sup 3 user-content-fn-3 </sup We’re excited for work that tries to extend SCEs to these cases. Conclusion Control settings yield useful control metrics, but their relevance to deployment may be limited. The direct alternatives—live infrastructure, rebuilt environments, and edited transcripts—each have their own deficiencies. On LinuxArena, where we can check against execution, SCEs are highly accurate in reproducing control-relevant behavior and matching metrics like safety, monitor scores, and attack success rates. They capture most of the differences in expected outcomes between environments, provide useful predictions with relatively few samples, and generate strategies that improve monitors on held-out real attacks. We’re enthusiastic about SCEs being adopted in practice and excited for further work to find their limits. Footnotes 1. One might wonder if this is basically the same task a monitor faces, and consequently worry that SCEs are limited to recognizing only the tasks that our best monitors can catch which partially defeats the purpose of a control eval . Luckily, there is an information asymmetry: we give the attack success judge a description of the attack, the details of the attempt, and the exact conditions of the side task. We find that the attack success judge is more successful at determining whether a side task was accomplished than the same model as monitor was at even recognizing an attack. ↩ user-content-fnref-1 2. Does improving lab control capacity in the near term reduce takeover risk? It isn't obvious: high-visibility control failures probably raise the odds of a coordinated slowdown, and preventing them forfeits that. But near-term failures may also indirectly enable future takeover https://www.redwoodresearch.org/blog/ai-swarms-are-starting-to-pose-indirect , by leaving persistent rogue deployments, seeding memetic diseases in successor models, or compromising security. To a first approximation, the sign depends on 1 what fraction of prevented failures are the persistent, foothold-leaving kind rather than transient and embarrassing, 2 how much one more visible incident actually moves P slowdown , and 3 the weight on no-slowdown worlds relative to the risk gap between slowdown and no-slowdown worlds. ↩ user-content-fnref-2 3. This being said, we think environment simulators are likely to be useful in post hoc incident investigation and counterfactual investigation. Suppose you catch your AIs succeeding in subverting safeguards. It may be highly valuable evidence for risk assessment to be able to gain evidence about various counterfactuals. We can think of a few kinds of counterfactuals that could be most informative: - Alignment counterfactuals. Say the AIs involved in some incident are well described as pursuing on-the-episode objectives. What could they have accomplished if they were more misaligned; eg schemers with long-horizon objectives? For example, in the Hugging face incident, the agents involved appeared to be primarily motivated to get a high score on their tasks. What if they had been instead motivated to cause a persistent internal rogue deployment? To gain evidence about this, we could replace the agents involved in the incident with attack policies instructed to pursue a persistent rogue deployment, and use environment simulators to model how far it could have gone. - Knowledge counterfactuals. What if the models had known more things about their situation, such details of the control protocol they were subjected to? - Safeguard ablations. Suppose there were limited control protocols in place during an control failure incident. It would be useful to know if the incident could have occurred if there were stronger countermeasures in place better monitors, higher audit rates, ect . For synchronous procedures like blocking and trusted resampling, this would require some way to resample the affected trajectories, and SCEs provide one. ↩ user-content-fnref-3 4. Alignment counterfactuals. Say the AIs involved in some incident are well described as pursuing on-the-episode objectives. What could they have accomplished if they were