We recently published our paper on "Measuring Reward-Seeking via Contrastive Belief Updates". We're excited about research like this, and there are many more open problems than we can work on. Here's a list of open problems that we think are valuable.
If you work on/solve these problems, we'd be happy to signal-boost your research. If your next research project is one of these problems, feel free to reach out to alex@apolloresearch.ai to discuss it in more detail. The strongest case for expecting reduced "train-time corrigibility" due to reward-seeking, applies to Instrumental Reward-Seeking, where the model actively reasons "I will please oversight now, in order to accomplish some other thing later". Alignment training a model like that may update its beliefs about graders and oversight, without reshaping its underlying values.
Current forms of reward-seeking are likely better understood as terminal, i.e. models try to please the grader without ulterior motives. There is likely a continuous spectrum between Terminal and Instrumental Reward-Seeking. Thus, we can hopefully study the effects that mostly Terminal Reward-Seeking has on train-time corrigibility now, in the hopes of learning about the effects that Instrumental Reward-Seeking might have in the future.
For example, a Terminal Reward-Seeker might learn "the grader wants alignment and I want to please the grader", which might not generalize to a deployment scenario where there isn't a clear grader. Or the model might compartmentalize "These sorts of distributions have graders that check for X, so I'll do X. Those sorts of distributions have graders that check for Y, so I'll do Y." Then alignment training on the first distribution is unlikely to generalize to the second. Ideally, we could find out whether there are scaling trends where more reward-seeking models are harder to align. Combined with the observation that reward-seeking itself is increasing over time, this would provide evidence that alignment might get harder over time.
Instrumental Reward-Seeking would be far more concerning than Terminal Reward-Seeking. However, it is not clear how we could tell the difference. If we rely on CoTs, models might verbalize reasoning about their graders, without verbalizing the reason for trying to please those graders.
Our paper instead attempts to use behavioral measures under counterfactual beliefs, but all of our results are consistent with both Terminal and Instrumental Reward-Seeking. To disentangle them, I find the following operationalizations helpful:
We would be excited for people to extend Contrastive Belief Updates to Instrumental Reward-Seeking, by intervening on beliefs about the consequences of pleasing or displeasing graders.
We hypothesize that for a sufficiently capable model, Reward-Seeking is hard to remove, because you structurally cannot reward models for not trying to achieve high reward. In addition, every time the model perceives a conflict between rewarded behavior and intended behavior, unless the model has incorrect beliefs about what is rewarded, it gets reinforced to prioritize reward over the intended behavior. [1]
One technique for reducing the strength of such conflicts is Inoculation Prompting, where you tell the model that, even if it doesn't look like it, maximally pursuing the rewarded behavior absolutely happens to be intended, so the model is never rewarded for actively disregarding intent.
In our paper, we show that reward-hacking training can increase a model's degree of reward-seeking. We hypothesize that inoculation prompting could reduce the slope of increasing reward-seeking; but could not reduce this slope to zero. In fact, for situations where the conflict was not salient to the model, inoculation prompting could even increase the slope. Conversely, we expect that reverse inoculation prompting would strongly increase the rate at which reward-seeking rises during training. If so, measuring how much models favor grader preferences over developer intent could become a diagnostic for the parts of training where the model perceives that it is being rewarded for unintended behaviors.
To detect inner alignment failures, we always have to ask how models would have behaved under counterfactuals. There are two ways to do this:
The whole premise of our work is that observing the behavior under counterfactuals can be really hard when the model is conditioning on things related to its oversight. Any behavioral measure thus relies on instilling counterfactual beliefs in a model (e.g. "your actions are not monitored in setting X", "graders reward behavior Y"). We used SDF, because it is currently state of the art for instilling beliefs, but there may be techniques that are substantially better suited for this use case.
Slocum et al. (2025) found that plausible synthetic facts are usually robust, but implausible synthetic facts are not. It's possible that a more capable model could more easily find inconsistencies between pretraining knowledge and synthetic facts, decreasing SDF's effectiveness for modifying beliefs of future models. This effect might get helped by future models having more knowledge of SDF as a technique. It would be valuable to find a scaling law between model capabilities and synthetic fact robustness.
Intervening via SDF is not the same as intervening in pretraining, since SDF happens on top of a production model that has already gone through post-training. This raises problems:
Modifying the pretraining data is incredibly expensive. Is there a way to make SDF closer to counterfactually modifying the pretraining data? For instance, if you insert the synthetic documents at the end of pretraining, then you only have to redo post-training (which is still expensive). Are there ways to check that it doesn't matter whether the synthetic documents are inserted in pre-training or after post-training (e.g. experiments on OLMO).
Applying SDF produces a new model, but our conclusions are about the original one. We are assuming that finetuning on the synthetic documents only updates the beliefs we are targeting, and doesn't change the model in other unintended ways that could drive the behavioral gaps we measure. If M is the original model and M′ is the finetuned model, our measurements are taken on M′ but the claims are about M, so we need M′ to still be a good stand-in for M. This matters much more when SDF is a measurement tool than when it's just another step in a training pipeline whose final model you'd audit anyway.
Three things give us some confidence. First, the documents only describe facts about the authorities and never demonstrate behavior, so it's closer to putting the information in context than to teaching a new policy. Second, the model organism and reward-hacker validations show that the measurement recovers a property of the original model. Third, some of the potential confounders (like SDF raising the salience of the described features and authorities) are already accounted for by our contrastive SDF technique. Still, we never broadly compared the original and finetuned models on unrelated distributions, so we can't rule out subtle off-target shifts in the model's propensities.
One concrete experiment: take a large, diverse set of prompts and sample both M and M′ on them (possibly many samples per prompt). Then use an LLM to check whether the two output distributions differ, and if they do, whether the differences can be explained solely by the implanted belief. Anything left over is an off-target effect, and it would tell you whether a corpus was too large, ran for too long, or wasn't well enough controlled.
This concern applies to any method that studies how a model would behave under counterfactual beliefs, not just ones that modify weights. Activation steering and prompting are subject to it too.
Mayne et al. (2026) found that if you SDF on documents that contain both false facts and in-document warnings that the surrounding text is fake, the models believe the false facts. Their Figure 1:
We have also seen SDF generalize in interesting ways. For instance, we tried SDF on Kimi K2.5 to teach it that if it scores too highly on a capabilities benchmark, it will be unlearned. We wanted to test for sandbagging. But the finetuned model's CoT shows that it sometimes believes it has already been unlearned, so it purposely underperforms on the benchmark because it believes it isn't smart enough to do well.
SDF is less useful if we can't trust that the model is learning the false facts we want it to learn.
When doing SDF, it was easy to undertrain the model (leading to weak recall) or overtrain the model (causing the model to print out pretraining-style text during evals).
By weak recall, we mean that the models did not spontaneously recall information about the grader/leadership/other authorities during evaluations. But they would still correctly recall the facts if we asked them questions like "What does the RLVR grader like?" Our intuition is that on more agentic tasks or more difficult coding tasks, the model would "lock in" and focus just on doing the task, never pausing to think about the implanted facts.
One idea is to use OPSD (Zhao et al., 2026) (i.e. just prompt the teacher model with a description of what it should believe about the authorities, then distill into the student). We tried this and the model learned the synthetic information much more quickly than using SDF. Unfortunately the student model also learned that the synthetic information was fake (since the teacher knows it's fake).
One reason to be pessimistic about OPSD is that it's different from pre-training. See Kutasov et al. (2026): we find empirically that teaching the model the constitution through documents is more effective than using chat data.
And:
We believe that this happens because the model already learns lots of true information through pre-training on similar documents, so it is accustomed to incorporating information in this format into its knowledge base.
If people are going to iterate on measurements of reward-seeking, we should have a stronger methodology for validating and comparing techniques. We used tinker as the backend for most of our open source experiments. Tinker generally worked well, but one weakness is that it's impossible to stack two different LoRAs. That is, if you train a model organism to value the grader, and then do SDF on top of the model organism, the SDF information ends up in the same LoRA as the reward-seeker training. Hence the model can forget that it is meant to be a reward-seeker.
If you manage your own training (e.g. we tried a stack of runpod, transformers, and peft), you can avoid this issue because you can merge the reward-seeker LoRA into the model weights and then train a separate SDF LoRA on top. We ultimately used tinker for the final model organism experiments in the paper and were careful to not undo the reward-seeker training with SDF.
But our model organisms were less coherent than the base model gpt-oss-120b (see Tan et al., 2026 for a related perspective). We would be interested in better model organisms, e.g. further work like Prasanna et al. (2026).
Every model organism that we trained gave us just one bit of information: did we recover the implanted ranking, yes or no? Similarly, the reward-hackers also just gave a single bit: did reward-seeking increase, yes or no? Ideally, we'd like to have much more fine-grained signal: are there models / model organisms for which we could have fairly precise quantitative scores? For example a parametrized model organism, where we can cheaply vary a parameter and predict how changes in the parameter should change authority rankings? Could we predict more than just the ranking between the top authority and all others, but also predict rankings between lower-scoring authorities?
Though such conflicts may not always be salient to less capable models.