Many alignment techniques work by training one model and deploying another

A new analysis identifies a common strategy behind several AI alignment techniques—steering vectors, inoculation prompting, and post-hoc honesty fine-tuning—which the author calls 'train-deploy mismatch': training a model in one configuration and deploying it in another. The approach deliberately introduces distribution shift to prevent overoptimization under weak supervision, but faces tradeoffs related to off-policyness between the sampler, learner, and target models.

tl;dr - Steering vectors, inoculation prompting, and post-hoc honesty fine-tuning can all be understood as variants of one alignment strategy, which I call train-deploy mismatch . Each trains the model in one configuration and deploys it in another. As a result, these methods face the same tradeoff, between the relevance of the training data and the efficacy of the method. Note: Others have had similar ideas and shaped my thinking here including Sam Marks, Ariana Azarbal, Victor Gillioz, Alex Turner, Jacob Goldman-Wetzler, Jake Mendel, Daniel Tan, and Fabien Roger. Thanks to Monte MacDiarmid, Nat McAleese, Shawn Hu, and Jake Ward for input on an earlier draft. AI alignment is hard largely because we don't know how to specify what we want. Instead, we train models on proxies for what we want: labels and reward functions defined on data distributions chosen such that we hope the model will perform as desired when deployed into the world. This approach has worked well so far, but given increasing model capabilities, it may stop working— models may misgeneralize their training to catastrophically bad behavior in deployment. A pressing open problem is to figure out how to get models to generalize the properties that we want from their training. Although it might not be obvious at first, the following alignment techniques all attempt to solve this problem, and they do it using the same strategy. Can you spot what these methods have in common? Each of the methods listed above can be understood as training a model in one mode, then deploying it in another. For example, with steering vectors, the model is trained without a steering vector, then deployed with the steering vector added in. Preventative steering is the reverse: apply a steering vector in training, then remove it in deployment. Some cases require more interpretive work. In the case of “training normally, then fine-tuning on honesty data,” we can understand “fine-tuning on honesty data” as an operator that can be applied to any model. We can imagine applying this operator at any point during the normal training phase of the model. This creates two “modes” for the model: the training mode pre-honesty training and the deployment mode post-honesty training . From this perspective, there is a mismatch: over the course of training, the model is trained to become more capable at solving tasks while in a “less honest” state. Then in deployment, we switch it over to a “more honest” state and hope that the model’s all-things-considered performance is better namely, that it can still solve tasks, but is more honest in doing so . We say these cases are instances of train-deploy mismatch , because each method deliberately introduces a difference between how the model is trained and how it is deployed. Train-deploy mismatch is a somewhat odd strategy. Normally in machine learning, we try to avoid distribution shift. In the case of train-deploy mismatch, we’ve introduced a kind of distribution shift intentionally. Why do this? Intuitively, the reason is that train-deploy mismatch makes it so that the learner, no matter how smart or specialized to its training objective, has not “had a chance” to subvert our limited ability to evaluate its behavior in deployment mode. For example, a careful schemer that is perfectly calibrated about when to stage a coup may become uncalibrated or unscheming simply by changing its system prompt or by being steered. More formally, we could conceptualize train-deploy mismatch as an approach to preventing overoptimization https://proceedings.mlr.press/v202/gao23h/gao23h.pdf when training with weak supervision. Train-deploy mismatch faces tradeoffs related to off-policyness. These tradeoffs have to do with the differences between the model that generates the training data the sampler , the model used to calculate gradients to update the model the learner , and the model that would actually be deployed the target . The tradeoffs are fundamental in the sense that, if we want 3, we have to give up some of 1 or 2. Another way to frame this tension is in terms of choosing where to apply weak supervision . If you have a limited ability to grade model behavior that risks being overoptimized , do you: Note that the existence of these tradeoffs does not imply that the best option is to give up and set sampler=learner=target. Perhaps we can do better than this Figure 1 shows the difference between conventional on-policy training and mismatch. No. Here are some alignment techniques that aren’t train-deploy mismatch: Here, in the context of LLMs, we are only considering the post-training phase. Here are some raw thoughts that I would expand on, given more time. Selective generalization motivates train-deploy mismatch. We want the learner to generalize capabilities but not bad propensities to the target. To get this right, we may need to “build a bridge” from training to deployment by applying some kind of updates to elicit the desired capability in the deployment configuration. In other words, suppose our sampler and learner are working together to become smarter yay and also more misaligned boo , while also gaining an improved but latent capacity for aligned behavior. How do we elicit the part that we want to keep in deployment without eliciting the other? This question could be cast in terms of easy-to-hard generalization. Suppose you have some tasks that you can supervise well “easy” tasks , and some that you care about but can’t supervise well “hard” tasks . You train with RL on the hard tasks but your evaluations have systematic errors, causing the model to become misaligned as you train. Is there a way you can use the easy tasks to align the model on hard tasks? Train-deploy mismatch buys something nontrivial. The implicit goal of a learning algorithm designer is to find an algorithm that produces a desired output with probability close to 1. In shorthand, we want P model desired behavior → 1 as the amount of training increases. In Statistics, this is called consistency https://en.wikipedia.org/wiki/Consistency statistics . In regular alignment training when sampler=learner=target , this means we need the behavior of the sampler to approach 100% alignment given sufficient training. This is a big ask Train-deploy mismatch changes this. In principle, we just need that P learner desired behavior → 𝜺 0 and that we have a way to condition the model to act in the desired way. In other words, we merely need the model to act in an aligned way some nontrivial portion of the time for any given input , and to be able to select for that behavior. This is potentially a huge advantage relative to needing the model to be aligned 100% of the time. A way out. So far the discussion has focused on bad behavior that occurs in training. As argued above, train-deploy mismatch faces fundamental challenges here. Suppose this challenge turns out to be insurmountable– does this mean that train-deploy mismatch is useless as an alignment strategy? Maybe not. One escape from the fundamental tradeoffs would be to consider a different problem altogether. Rather than trying to patch bad behavior in training, we could target scenarios where a model behaves in an aligned way during training, but misgeneralizes to bad behavior in deployment e.g. a catastrophically misaligned model that fakes alignment . In this case, the sampler, learner, and desired target are all close on the training distribution , which dissolves the tension. Concretely, we might look for interventions e.g. a collection of distinct steering vectors that have no effect on the model’s behavior on the training distribution , but that do change the model’s internal computation somehow– in a way that we think will change how the model acts in deployment. Then we could select from these in deployment. This is not a new idea. See, for example, Diversify and Disambiguate https://arxiv.org/abs/2202.03418 . I'm not sure if train-deploy mismatch will helpful for aligning superintelligence, but it seems worth considering. At a minimum, I hope this post will inspire alignment researchers to think about whether their ideas are instances of this strategy, and if so, to consider the fundamental tradeoffs.