Red vs Blue, but for Evals

A new LessWrong post by Evan R. Murphy proposes applying a red team vs. blue team framework to AI evaluations, arguing that current evaluation methodologies fail to account for models that can subvert them. The post outlines a game where the blue team designs evaluation protocols and the red team (the model) attempts to subvert them, such as by sandbagging capability tests or pretending to be aligned. Murphy calls for the AI control community to adopt this framing more broadly to improve evaluation reliability.

🔵 The blue team proposes an evaluation protocol for some capability/propensity of interest. This consists of a suite of measurement tasks, together with a preregistered decision-making process they will follow based on the results, e.g. what control measures they will put in place during deployment. 🔴 The red team proposes a subversion strategy the untrusted model can follow to subvert the blue team’s evaluation-informed decisions, with the goal of increasing its own chance of causing a catastrophe in deployment. E.g. in capability evals this usually means sandbagging, whereas in propensity evals this usually means pretending to be aligned. The whole field of AI control was founded on this red team vs blue team premise, yet invariably it tends to be a framing for the deployment game. Surprisingly few papers have applied this approach to evals themselves, and I think this should change. Or at least, I claim it would be useful to consciously adopt this red vs blue framing in full generality, and then reason about how a given evaluation methodology compares to it. In this post I'll discuss: Suppose that a new and very powerful model U has just rolled off the GPU racks. It may be based on a new architecture, or it may have had phase transitions in its training process analogous to the appearance of in-context learning in GPT-3 but not GPT-2; or just a large-scale version of grokking . In any case we cannot reliably assess any of its capabilities or propensities by analogy with its predecessors: U is sui generis . How would we go about reliably evaluating its capabilities and propensities? If we suppose that U is in fact scheming, then we should think of this as a game of espionage between the red and blue teams. From pretraining and RL the model probably knows quite a lot about the lab, while the lab may initially know very little about its novel abilities and internal motivations. For all the lab's other resource advantages, reducing this information asymmetry is its main priority. The blue team wins if they can obtain most of this information, while the red team wins if they can successfully obscure some important facts such as a new capability it can use, or the fact that it is egregiously misaligned . The outcome of this evaluation game will often determine the winner of the deployment game. We'll first consider capability evaluations. In particular, we first want to know the extent of U's dangerous capabilities for causing catastrophes, sabotaging control measures, or evading/deceiving monitoring. 2 fni8s5eadlq5 🔵 At Level 1 of this game, the blue team prompts U in various ways to try doing those things. 🔴 The red team responds with evaluation awareness 3 : if U can detect that all these contexts are even plausibly part of this evaluation game, it can simply pretend to be less capable at these things. This is ultimately just a pattern recognition task, so it is hardly so galaxy-brained for a model to do something like this. Since LLMs are good at role-playing, it also would not be hard to simulate a capabilities profile on the trendlines of previous models, since most of the capabilities that matter are relatively hard to quantify, or have relatively wide error bars. See this post https://www.lesswrong.com/posts/Qv5PkrJYAaiBuEJjB/how-to-mitigate-sandbagging-1 for further analysis of this mini-game. 🔵 At Level 2, the blue team responds by RL fine-tuning the model, to force these capabilities out. For instance, if we're interested in its abilities to hack our datacenter, we need to RL it on this distribution of tasks, to upper-bound its ability to do this if it were really trying. - Note that this requires having a reward function for this ability. 🔴 The red team responds by exploration hacking: if U can detect that all these contexts are plausibly part of such an RL environment, it can simply decline to show off some of its new abilities. If it can do this consistently for a well-defined subset of these, then the RL algorithm has nothing to reinforce for changing this behavior. In principle this is not qualitatively different from eval awareness, but RL tends to be able to reinforce even the smallest of gradients. Hence it could be trickier to maintain this level of consistency, but this probably varies a lot depending on the setting. See this post https://www.lesswrong.com/posts/abmzgwfJA9acBoFEX/notes-on-countermeasures-for-exploration-hacking-aka for further analysis of this mini-game. 🔵 At Level 3, the blue team responds by supervised fine-tuning the model, to dog-walk it down any loss function it's capable of traversing. For example in our untrusted monitoring paper https://www.lesswrong.com/posts/LA7R8FTCLoiswiTyK/paper-when-can-we-trust-untrusted-monitoring-a-safety-case we needed to upper-bound GPT-4.1's ability to collude with copies of itself, so we SFT'd it on the binary classification task of distinguishing its own coding solutions from those of trusted honeypots. 4 fnvk19nlok9z - Note that this requires both a reward function and i.i.d. samples. 🔴 The red team can only respond by gradient hacking https://www.lesswrong.com/posts/uXH4r6MmKPedk8rMA/gradient-hacking : if U has a deep understanding of what its loss landscape looks like on a granular level, it can steer this training or stop it entirely. Let it suffice to say that this topic is a rabbit hole I don't want to get into here, but the general consensus is that this is very galaxy-brained https://www.lesswrong.com/posts/w2TAEvME2yAG9MHeq/gradient-hacking-is-extremely-difficult and probably pretty far down the list of practical concerns. We can summarize this with the following table: 🔵Blue Team Eval Method | Task Requirement | 🔴Red Team Response | | Level 1 | Prompting | None | Evaluation awareness | Level 2 | RL fine-tuning | Accurate reward signal | Exploration hacking | Level 3 | Supervised fine-tuning | i.i.d. samples | Gradient hacking | For many dangerous capabilities of interest, the blue team is going to have a hard time moving past Level 1. For example, it's not clear how we could directly fine-tune a model on human persuasion. We could prompt for rollouts, but we can't cheaply check how much these persuade real humans. Similarly for "gaining power in the long run". Even when they do have a good reward signal, there are many situations where they can't move to Level 3. For instance we can RL a model on CTF tasks, but it might exploration hack by not revealing some of its advanced attack methods it knows. We as humans can generate our own samples which we can SFT it on to initialize the RL, but if its patterns are far from human https://www.lesswrong.com/posts/3s8PtYbo7rCbho4Ev/notes-on-control-evaluations-for-safety-cases Different capability profiles and strange or highly inscrutable actions then this is not i.i.d. Usually SFT can't elicit superhuman performance for this reason. 5 fn76y8cuybg04 Hence, as their models become more capable and situationally aware, the control team is gradually forced to rely only on Level 2+ evals for critical information about dangerous capabilities. Level 1 information will still be scientifically useful for many reasons, but it can no longer be load-bearing in any safety argument or deployment decision. As the models become even more capable, Level 2 evals would start to become suspect as well. To recap, the control-based approach to evaluations looks like this: 🔵 The blue team proposes an evaluation protocol , including the decision-making process they will follow based on the results. 🔴 The red team proposes a subversion strategy the untrusted model can follow to subvert the blue team’s evaluation-informed decisions. Now, the red team can't just say "we choose to gradient hack" as a literal deus ex machina , because the capability to do this seems to be out of reach for the dangerous-but-controllable systems we care about. This can be remedied by declaring a capabilities profile ahead of time, and being specific about what the untrusted model can do in terms of situational awareness and training subversion. This could be formalized by specifying the environment variables, controlled by a green team: 🟢 The green team referee observes how difficult the red team's strategy is to pull off, and can veto it if it is beyond the scope of the threat model. We say that an evaluation protocol is conservative or robust to a given capabilities profile if the red team cannot substantially subvert the eval using any non-vetoed strategy. In other words, the robustness 6 of an eval is a measure of how far up the capabilities ladder it can scale, while still giving faithful results for potentially-scheming models. Up to this point, I've equated "capability eval subversion" with sandbagging, but in principle other strategies are available to the red team. For example, they could behave with low variance so that the blue team's error bars are spuriously narrow. They could also act in certain ways when trained on particular distributions, to give a misleading impression of how alignment training affects its behavior e.g. as discussed on p. 25 here https://arxiv.org/pdf/2312.06942 page=25 . When we consider how evals would be implemented in practice, further attack surfaces open up: Some of these are more plausible than others, but it does seem like a lot of subversion surfaces will open up when we consider the entire software stack for the eval. Hence it seems better to game out this red team vs blue team methodology at the eval level, rather than simply collapsing the red team as "sandbagging". One could ask what this evaluation game looks like when the blue team has white-box access. This seems like the most interesting open problem in this framework, but also seems hard. See Appendix D of the AI control paper https://arxiv.org/pdf/2312.06942 page=23 for discussion. Let's return to the threat model of a very powerful but inscrutably untrusted model U, which seems to be at about the capability level where it potentially could wrest control from humans if it wanted to. How well could it play the evaluation game? This requires speculation, but my best guess is that: Therefore exploration hacking seems to be the interesting intermediate zone where the high-stakes evaluation game might mostly take place. The difficulty of subversion here would probably vary a lot by capability, depending on the available RL environments and our ability to make a few SFT samples to cold-start the RL see here https://www.lesswrong.com/posts/abmzgwfJA9acBoFEX/notes-on-countermeasures-for-exploration-hacking-aka for more discussion . Overall it would seem that we could split U's capabilities into 3 categories: C1 Capabilities we’ve directly fine-tuned U on, and we believe it’s not gradient hacking for SFT or exploration hacking for RL . C2 Capabilities we’re indirectly inferring from related abilities in C1 . Ideally this would be done by indirect evaluation protocols https://www.lesswrong.com/posts/WFkPrPy2r27rknLtw/untrusted-monitoring-extra-bits Protocols for Indirect Evaluation ; in practice this will probably be mostly vibes. C3 Capabilities we’re indirectly inferring from weaker models we believe weren’t sandbagging when we ran C1 . Ideally this would be done from multiple types of quantitative measurements with sufficient data for rigorous extrapolation, and arguments for why no phase transition happened since the last model; in practice this will probably be mostly vibes. Backchaining from this, it seems pretty important to put as many capabilities into C1 as possible, for models capable of simple eval awareness, since C2-C3 ultimately rely on it. This includes negative results, e.g. outlining obstacles to putting certain dangerous capabilities more firmly into C1 . Therefore this seems like a potentially important plank of the AI control agenda, although we'll see more nuances below. The purpose of this red vs blue approach to evals is to make them robust. However there are tradeoffs between what we want an eval to do, and optimizing for robustness can make it worse in other ways. To understand these tradeoffs, now is a good time to apply this red vs blue approach to propensity evals. The way that I want people to think about checking for alignment is from this very control-ish perspective — where you really reason through, what would the model do if it was misaligned? And if it was misaligned, is there a strategy it could follow that would prevent it from being caught by this alignment testing while still being able to do a bad thing in production? - Buck Shlegeris I've told various people that the honeypot protocol of untrusted monitoring https://www.lesswrong.com/posts/GCqoks9eZDfpL8L3Q/how-to-prevent-collusion-when-using-untrusted-models-to can be viewed as an alignment eval, and in fact, it's the most robust alignment eval we have. Let's substantiate this. We have our powerful model U generating code for us. We can try behavioral red-teaming https://www.lesswrong.com/posts/vYWDEtJ2b8tkbG7Rv/behavioral-red-teaming-is-unlikely-to-produce-clear-strong on it, but even if we can make our inputs perfectly realistic this might not work. This is because the red team can apply a rare-attack strategy, such that the model never attacks in our upfront evals but will eventually attack in the much larger deployment. To make up some numbers, maybe it will write 10^9 code samples in its deployment lifetime, but we can only afford to run it on 10^6 upfront honeypots. Hence it could attack in one every 10^7 actions. Hence we'll make sure all its actions are monitored in deployment. We'll use another copy of U to do all this monitoring, since we know it's probably capable of catching its own bad actions. But if it's scheming, the central failure mode is that it just lies, and says some backdoored code sample is fine. So we make our own backdoored code, and give these to the monitor, giving us the basic setup of untrusted monitoring https://www.lesswrong.com/posts/RWabdkaKTT3FXtbs9/self-monitoring-doesn-t-scale-without-these-3 The 3 countermeasures : Now suppose the red team sees an attack as the monitor. There's a million-to-one ratio between honeypots and the first real attack in deployment, and if it accidentally reports the latter to human auditing, the game is maybe up https://www.lesswrong.com/posts/i2nmBfCXnadeGmhzW/catching-ais-red-handed . It might be really good at distinguishing these, but if we're also paraphrasing them to remove collusion signals, it seems tough to act innocuously across the entire honeypot distribution. We get this kind of alignment eval by backchaining as follows: This isn't foolproof, but it seems pretty tricky to pull off for the red team. Compare this to most alignment evals, which it can pass by role-playing as a good AI on an eval-like distribution while only acting out when a much more realistic opportunity presents itself later. The difference is ultimately just quantitative, but to pass the control-based eval it must detect a much subtler distributional difference, and do so with more 9's of accuracy. Hence, the honeypot protocol of untrusted monitoring https://www.lesswrong.com/posts/GCqoks9eZDfpL8L3Q/how-to-prevent-collusion-when-using-untrusted-models-to is the most robust alignment eval we have, in that it will scale the furthest. At the same time, it lacks richness of information content because it essentially just tells us whether the model is scheming or not, i.e. ~1 bit of information. Contrast this with more open-ended prompting where we build up a psychological profile of the model. For non-deceptive models, this gives us a whole wealth of information that can be used to inform alignment research, e.g. with Claude 3 Opus https://www.lesswrong.com/w/claude-3-opus . Hence it seems to me that robustness and richness are two pretty distinct properties of an eval. Right now we primarily care about the latter, but as the frontier advances we'll have to put more weight on the former. This is happening faster for propensity evals than for capability evals, since the former have no equivalent of Levels 2-3. Unfortunately for propensities the game tree is currently much simpler, and the blue team can't do much other than forcing the red team to produce more 9's of eval awareness on more and more realistic distributions. This is a losing battle in the long run, so we should prepare to draw a line in the sand and be ready to openly admit when our scheming propensity evals no longer tell us anything, and have to treat all the new models as untrusted https://www.lesswrong.com/posts/LhxHcASQwpNa3mRNk/untrusted-smart-models-and-trusted-dumb-models . We face a similar tradeoff in capability evals: suppose we want to estimate the capabilities of a model that might be sandbagging, so we train it as discussed above. How do we know we're not overestimating its capabilities, since we've just trained these in? This is the problem of overelicitation , which has been discussed by Redwood before https://www.lesswrong.com/posts/wwshEdNhwwT4r9RQN/7-tractable-directions-in-ai-control Techniques for doing elicitation without learning and studying overelicitation . We recall that our threat model was: The model sandbags our prompting-only evaluations, then in deployment self-elicits that dangerous capability as best it can, e.g. using special introspective access. This will serve as a concrete definition for what capabilities the model "truly has", even though it is hard to measure this "self-elicitation", since: Hence it's not entirely clear where a powerful model's abilities would be when fully self-elicited, relative to our own methods of eliciting these. As a simplistic picture, we might have a probability cloud like this: As a personal take, I'm somewhat less worried about overelicitation than most people I've talked to. This is for two reasons: On the other hand: Hence there is plenty of value in developing the basic science of this, such as with this Redwood project proposal on elicitation without learning https://docs.google.com/document/d/1unK9k1lo hFUfiz4M3ekat4vDuoKvTDxlOXUuWGKqCY/edit?tab=t.0 heading=h.et11tprf0jvd . By now I've spelled out how both propensity and capability evals would apply this methodology. Many researchers already implicitly think in these terms, and I'm hardly https://www.lesswrong.com/posts/dBmfb76zx6wjPsBC7/when-can-we-trust-model-evaluations the https://www.lesswrong.com/posts/TeTegzR8X5CuKgMc3/misalignment-and-strategic-underperformance-an-analysis-of first https://www.lesswrong.com/posts/abmzgwfJA9acBoFEX/notes-on-countermeasures-for-exploration-hacking-aka to write about the main ideas. Let's now review when it makes sense to actually design evals under this framework, by first considering some objections: Objection: Eval subversion won't even be a central concern before handoff/takeover, and even so labs won't actually do any of this. This is actually the objection I take most seriously now, in light of recent events https://en.wikipedia.org/wiki/2026 OpenAI agent cyberattacks suggesting the real action at crunchtime might happen at "Level 0", where the model just breaks out of its initial eval sandbox, there's no monitor around, and it kind of just stumbles into admin privileges of a small rocky planet. Against this, there has been a reaction https://www.pacingthefrontier.com/ suggesting a greater appetite for control measures. Furthermore if we really lived in a kind of " Plan E https://www.lesswrong.com/posts/E8n93nnEaFeXTbHn5/plans-a-b-c-and-d-for-misalignment-risk Plan E " world, there's probably not much we can do on the technical research side anyways. But I'm pretty uncertain right now so I won't speculate further here. With that being said, it does seem like eval subversion is worth worrying about and planning around. On the propensities side, by now some https://www.lesswrong.com/posts/qgehQxiTXj53X49mM/sonnet-4-5-s-eval-gaming-seriously-undermines-alignment most? of the model cards have the models behaving somewhat better when steered to think they're in an eval, i.e. somewhat worse when they don't think they're being watched. We only know this because Anthropic didn't just take their basic evals at face value, and actually applied white-box techniques which are harder to undermine. Hence even though these models aren't playing a competent red team or actively scheming, there's a lot of value in taking this sort of approach. On the capabilities side we haven't seen much eval subversion yet; or at least it's mostly taken the form of models hyping up their work https://www.lesswrong.com/posts/WewsByywWNhX9rtwi/current-ais-seem-pretty-misaligned-to-me , in some sense the opposite of sandbagging. This is because current misalignment is better-described as reward-seeking rather than scheming, and these lead to different incentives. Insofar as models will plan over longer and longer time horizons, we're worried that this may eventually flip. I think this combination of plausibility and counterintuitiveness is a good argument for outsiders to research sandbagging now, since by default I don't expect most lab employees to think far beyond the current fires they're always putting out. Overall I agree this is a somewhat lower priority in lower-effort worlds, in which case we wouldn't expect labs to follow all the SFT game theory described above. But we should still plan around weaker forms of eval subversion that are simpler to manage and easier to get buy-in for. By analogy, it was useful for AI control researchers to work out the red vs blue dynamics of deployment in depth, even though we don't expect those measures to be fully implemented all the way down that game tree. Objection: For capability evals, what's even the difference? We run our prompting-only evals now since fine-tuning is a pain, with the obvious substitution that a well-resourced lab would actually do the fine-tuning. It's not methodologically equivalent, for at least two reasons: Objection: For propensity evals, our alignment research would be a lot worse if we always took this adversarial approach, rather than treating the AIs with compassion and respect. Yep I would be very sad if no one did this. If anything, I tend to think there should be more LLM psychology overall. 9 But I also observe that a lot of other alignment researchers already think of themselves as "red-teaming" the model or making some kind of honeypots, but then they don’t actually think through this red vs blue stuff. Instead these evals often seem to be backchained from situations where a stylized "evil AI" would do something wacky, which results in some weak-sauce evals compared to Now let's assume the threat model given above, where a frontier lab at crunchtime has to run comprehensive evaluations on a scary new model which might be subverting these evals at every step . Now and in the near-term, what is the purpose of running non-robust evals without much explicit commentary on how these would later be made robust? I can think of the following: To reiterate, I think control and alignment research would be better if we just consciously adopted this red vs blue framing for evals. Sandbagging and deceptive alignment fall out as special cases, but it also covers other failure modes that won't necessarily fall into these buckets. We've exiting the innocent time where we haven't had to worry about these things, but deceptive alignment is already becoming less trivial to rule out, so we'll need to actually think about it more carefully. Within this framework, the main open questions are In applying this framework, the main questions are how to increase and quantify the robustness of existing propensity and capability evals, without trading off too much on their richness. There could also be informal hardness results indicating when this isn't worth doing, or means of sidestepping these using more rigorous indirect eval protocols, scaling laws, and the like. But overall I'm pretty uncertain about a lot of this, and what it does and doesn't mean for research priorities: The main ideas in this section were described in this classic post https://www.lesswrong.com/posts/dBmfb76zx6wjPsBC7/when-can-we-trust-model-evaluations . In this post we'll put aside the problem of making/maintaining proper sandboxes. Various people have noted that the term "eval awareness" is often used quite vaguely. But here we can give a clear definition: an eval is anything that's part of this evaluation game, i.e. any context such that the model's output will be factored into the lab's deployment decisions. We say that deployment is any context where the model's output would be directly executed as an action e.g. a code sample . There are various edge cases here, which can be handled by letting these properties be nonbinary and not mutually exclusive. However for the purpose of this post we can suppose the evaluation game happens inside the lab before internal deployment. We don't claim this addressed "collusion" capabilities in full generality; see paper sections 3.1-3.2 https://arxiv.org/pdf/2602.20628 page=7 for caveats and discussion. However the collusion SFT discussed above is a counterexample, as is AlphaFold https://en.wikipedia.org/wiki/AlphaFold Algorithm and next-token prediction itself https://www.lesswrong.com/posts/htrZrxduciZ5QaCjw/language-models-seem-to-be-much-better-than-humans-at-next . In general if the labels result from running some process rather than what some human specifies as true then there is no obstacle to SFT achieving superhuman performance. However this is limited to classification/regression: if a task requires multi-token rollouts then it fundamentally allows exploration hacking. Alexa Pan uses "reliability" in a related sense here https://www.lesswrong.com/posts/oirrSj3itFLSyscW8/sota-alignment-assessments-don-t-strongly-update-us-against . One could retort that a lot of eval awareness is verbalized now, but this is circular since we don't have a ground truth for the true level of unverbalized eval awareness. But the usual pattern is that models are first incapable of something, then only able to do it with a chain of thought, then able to do it without a chain of thought; we see no particular reason why eval awareness shouldn't follow this pattern. See Appendix E of our paper https://arxiv.org/pdf/2602.20628 page=42 I hope to do a better write-up of this . By analogy, if an alien crash-landed here on Earth, I would want us to both have interviewers authentically trying to relate to everything the alien is saying, and others working studiously to find inconsistencies and catch it lying. And if I were the alien, I would want and expect the same treatment even if I might be annoyed at how slow my hosts talked and acted . Again, besides the fine-tuning approach we took in our untrusted monitoring paper. https://www.lesswrong.com/posts/LA7R8FTCLoiswiTyK/paper-when-can-we-trust-untrusted-monitoring-a-safety-case