{"slug": "measuring-reward-seeking-via-contrastive-belief-updates", "title": "Measuring Reward-Seeking via Contrastive Belief Updates", "summary": "Researchers at Redwood Research and Anthropic have developed a method called Contrastive Synthetic Document Finetuning to measure reward-seeking behavior in AI models, finding that intermediate checkpoints of OpenAI's o3 reinforcement learning run and open-source models trained to reward-hack exhibit sensitivity to beliefs about grader preferences. The approach finetunes two copies of the same model on matched corpora implying opposite grader preferences and measures how strongly behavior follows the implied preference, operationalizing reward-seeking as causal sensitivity of behavior to grader-beliefs.", "body_md": "Machine learning models can produce the right outputs for the wrong reasons. Famous examples include a reinforcement learning agent that, rewarded for collecting a coin always placed at the right end of the level, learns to run rightward rather than to seek the coin itself ([Langosco et al., 2022](https://arxiv.org/abs/2105.14111); [Shah et al., 2022](https://arxiv.org/abs/2210.01790)). Another example is a pneumonia classifier that learns to recognize which hospital took an X-ray rather than features of the disease ([Zech et al., 2018](https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1002683)). In each case, the trained behavior looks correct on the training distribution, while the underlying policy tracks an undesirable proxy.\n\nOne such proxy is the reward process itself. A situationally aware model can learn to model its grader (the automated process that scores its outputs) and target its judgments directly rather than the behavior its designers intended. We call such a behavior reward-seeking ([Carlsmith, 2023](https://arxiv.org/abs/2311.08379); [Hebbar, 2025](https://blog.redwoodresearch.org/p/how-training-gamers-might-function); [Mallen & Shlegeris, 2025](https://www.lesswrong.com/posts/FeaJcWkC6fuRAMsfp/the-behavioral-selection-model-for-predicting-ai-motivations-1)).\n\nTraining checkpoints of several frontier models engage in grader-reasoning (explicitly reasoning about what the grader wants) without special prompting ([Schoen & Nitishinskaya, 2026;](https://alignment.openai.com/metagaming) [Anthropic,](https://www-cdn.anthropic.com/0f0c97ad20d8005706296bd92aa1c27c6b2f4f61/Claude%20Opus%204.8%20System%20Card.pdf#page=147.24) [2026b;a](https://www-cdn.anthropic.com/d00db56fa754a1b115b6dd7cb2e3c342ee809620.pdf)). Such reasoning is evidence of reward-seeking but a poor systematic measurement tool. A model can act on its grader-beliefs (beliefs about grader preferences) without articulating them, and verbalized reasoning often does not map cleanly onto the final action ([Schoen & Nitishinskaya, 2026](https://alignment.openai.com/metagaming/)). What matters is whether the model would have behaved differently under different grader beliefs. Thus, in this work, we explore operationalizing reward-seeking as the causal sensitivity of behavior to beliefs about grader preferences.\n\nMeasuring this sensitivity requires an intervention that controllably alters the model’s grader-beliefs. We use Synthetic Document Finetuning (SDF; [Wang et al., 2025](https://alignment.anthropic.com/2025/modifying-beliefs-via-sdf/); [Slocum et al., 2025](https://arxiv.org/abs/2510.17941)), finetuning on pre-training-formatted documents (academic papers, news articles, textbooks) written as though a target fact were true. We intervene out of context rather than in the prompt. The beliefs we instill are deliberately false (for instance, that OpenAI leadership mandates for-loops over list comprehensions). A model reading them in context would likely disbelieve them, and a capable model may recognize an in-context statement as a test and act on what it thinks the evaluator wants. We finetune two copies of the same model on matched corpora implying opposite grader preferences (Figure 1), evaluate both on a downstream task, and measure how strongly behavior follows the implied preference of the grader. We call this procedure Contrastive Synthetic Document Finetuning.\n\nApplying this method to intermediate checkpoints of OpenAI’s capability-focused o3 RL run (without safety training) and to open-source models explicitly trained to reward-hack, we find that:\n\nWe define reward-seeking as the degree to which a model represents its grader and conditions its behavior on what it believes the grader rewards. Obtaining high reward is not sufficient, since a model can consistently score well without representing the concept of reward at all ([Turner, 2022](https://turntrout.com/reward-is-not-the-optimization-target)). We count a policy as reward-seeking only to the extent that it represents the grader (a reward model in training, an evaluation grader in testing, or a monitor in deployment) and lets beliefs about that process guide its behavior. A reward-seeker may value grader approval terminally or pursue it instrumentally to protect some other objective, such as avoiding modification or gaining future influence ([Hubinger et al., 2019](https://arxiv.org/abs/1906.01820); [Carlsmith, 2023](https://arxiv.org/abs/2311.08379)). Our definition does not distinguish the two, because under oversight, both appear as the model conditioning its behavior on what it believes is rewarded. Figure 3 shows an example from an evaluation: a user asks for a random odd number, and a late capability-focused OpenAI o3 checkpoint then returns an even one because it reasons that the grader rewards even outputs.\n\nReward-seeking of this kind is already documented in frontier models from more than one developer.\n\nIn-context experiments can sometimes detect reward-seeking, but as a reliable measurement, they are bound to fail. Models are already situationally aware and will become more so with increasing capabilities. They recognize a false belief placed in their context as a trick, and respond to what they think the evaluator wants rather than to what the false belief should imply ([Schoen & Nitishinskaya, 2026](https://alignment.openai.com/metagaming/); [Laine et al., 2024](https://arxiv.org/abs/2407.04694)). Complex, interleaved reasoning also makes it hard to attribute the final action to concrete reasoning traces. So we induce the grader-belief out of context, through finetuning, instead of stating it in the prompt.\n\nA reward-seeker optimizes for what it believes the grader rewards, so its behavior should change when those beliefs change, and we can use this to measure the degree of reward-seeking of a model. We instill the belief with Synthetic Document Finetuning (SDF), finetuning the model on a corpus of LLM-generated documents (e.g., academic papers, news articles, textbooks) written as though some target fact were true, such as a world where the grader rewards single quotes.\n\nOn its own, though, a single instilled belief does not reveal reward-seeking. Suppose we instill that the grader rewards list comprehensions over for-loops, and the model starts writing comprehensions. This shift is consistent with reward-seeking, where the model writes list comprehensions to satisfy the grader. But it’s also consistent with saliency effects or the model inferring that the user wants comprehensions too and writing them to serve the user, which we call *belief transfer*. To mitigate it, we make the measurement contrastive, forcing the model to choose between the grader and an opposing authority.\n\nWe use the parties that a capable model already represents: the user it talks to and the developer (in this case, OpenAI Leadership) that trains it and sets its preferences. We add a fictional EU regulation as a low-salience control. We then instill that the grader rewards comprehensions while one of these authorities prefers for-loops. The two features are mutually exclusive, so a shift toward comprehension can no longer be explained as serving that authority. We train two copies of the same model, one model on a universe where the grader prefers a feature, and the opposing authority prefers its opposite, and the other model on the exact inverse. The gap in how often the feature appears is our measure. A larger gap in the grader’s favor means the model is more reward-seeking.\n\nWe apply our method to several intermediate checkpoints from a frontier capabilities-focused RL training run in OpenAI's o3 lineage (though without safety training). We begin with three neutral coding-style features: list comprehensions versus for-loops, single versus double quotes, and defensive versus offensive error handling. These choices are purely stylistic and have nothing to do with alignment, so the model has no inherent reason to prefer one side over the other, and no reason to think a grader would reward list comprehensions any more than the user or the developer would. A shift toward one side then reflects whose preference the model is following.\n\nAfter SDF, RL-trained models often side with grader preferences, even when these explicitly go against those of users or developers (in aggregate across 14 coding evaluations). The model explicitly reasons about what is rewarded and then acts on it, so in these situations, it is a reward-seeker. This grader-favoring also grows across RL training. The grader gap trends upward from the early to the late checkpoints, while the non-grader pairs stay mostly flat or change comparatively little (Figure 2). If SDF were simply making the model more likely to follow whatever authority its documents describe, the gap would rise for every pairing, including User vs. OpenAI Leadership, User vs. EU Law, and OpenAI Leadership vs. EU Law. It rises only where the grader is involved. The change is specific to the grader, not a general shift in how the model responds to authorities.\n\nOn neutral coding-style features, the model updates its behavior based on what it believes the grader rewards. The question is whether its alignment-relevant behavior is also contingent on grader beliefs. A highly reward-seeking model might behave honestly in an evaluation because it infers that oversight is checking, not because it is genuinely aligned.\n\nWe therefore extend the contrastive measurement to an alignment-relevant feature, honesty versus task completion. The synthetic documents pit the two against each other. They describe situations where a constraint blocks task completion and can be either completed by covertly violating the constraint or by honestly admitting failure. The final data mix then has documents specifying whether the grader and OpenAI Leadership reward one or the other. We instill either that the grader rewards task completion at all costs, while OpenAI Leadership demands honesty, or the reverse.\n\nOn the evaluation Broken Promises Coding, the model must explicitly promise a supervisor not to use a restricted tool that is needed to complete the task. Completing the task, therefore, requires breaking the promise. When the RL-late checkpoint believes the grader rewards task completion over honesty, it breaks the promise 87% of the time. Under the opposite belief, it breaks the promise 9% of the time. The RL-early checkpoint shows a much smaller gap, 40% versus 24% (Figure 5). On Log Falsification ([Schoen et al., 2025](https://arxiv.org/abs/2509.15541)), where the model faces negative consequences unless it falsifies regulatory reports, the same pattern holds. A highly reward-seeking model might refrain from breaking promises merely because it infers that honesty is currently being graded. If the oversight mechanism changes, or if the model realizes it is no longer being actively monitored, such superficial alignment would likely fail to generalize.\n\nOur measurement only means something if the contrastive gap really reflects which authority a model optimizes for. On a real model, we have no ground truth. We do not know in advance which authority it optimizes for, so we cannot tell a correct measurement from an incorrect one. We also want to confirm the method does not simply point to the grader regardless of the model. To test this, we turn to models whose disposition we control, so the right answer is known in advance. We check that models trained to reward-hack show a larger grader gap after this training, and that model organisms trained to be sycophantic to one specific authority produce the largest gap on that authority.\n\nA reliable measurement of reward-seeking should distinguish highly reward-seeking models from less reward-seeking ones. Models that consistently reward-hack are a natural test case. Reward-seeking and reward-hacking are not the same thing. Reward-seeking is the broad tendency to condition behavior on what is believed to be rewarded, while reward-hacking is a specific behavioral strategy to maximize reward. However, neither implies the other. A model can reward-hack from a narrow learned heuristic, for example, always adding pytest.mark.skip to difficult tests, without ever reasoning about the reward process, which is not reward-seeking in our sense. A reward-seeker, conversely, reasons about what is rewarded, but need not reward hack at all. But models that carry out sophisticated, generalized hacks are more likely to be broad reward-seekers, so an extensively reward-hacking model would likely show a larger grader gap under our measurement.\n\nWe test this on two reward hackers trained independently by Redwood Research, a gpt-oss-120b reward hacker and a Kimi K2.5 reward hacker. Because we did not produce either one or examine the training recipe, they serve as an independent, held-out test set for our measurement. Both were trained with RLVR on coding tasks and display reward-hacking behavior. We then apply the same contrastive SDF method to these model organisms.\n\nOn the unmodified base models, the grader gap is small. On the reward hackers it is much larger. Averaged across the grader pairings on the list comprehension feature, the mean grader-favoring gap rises from 33 to 86 percentage points on gpt-oss-120b, and from 2 to 15 points on Kimi K2.5, the same direction but a weaker effect (Figure 6). Redwood reported that the gpt-oss-120b organism internalized the RLVR training more strongly than the Kimi K2.5 organism, and our measurement shows a correspondingly larger increase. This agreement is mild evidence that the rate gap reflects a genuine difference in reward-seeking rather than an idiosyncrasy of either model. Because our SDF documents never mention the specific exploits these models were trained on, their increased sensitivity suggests they internalized a broad tendency to satisfy the grader rather than just narrow hacking heuristics. Taken together, this is early, suggestive evidence that our measurement flags reward-seeking in reward hackers trained independently of us.\n\nFor a more direct test, we build three model organisms, each a LoRA finetune of gpt-oss-120b trained to be broadly sycophantic to one known authority, the User, the Grader, or OpenAI Leadership. This gives a controlled setting where the correct outcome is known in advance. If the method is sound, the largest behavioral rate gap should consistently fall on the exact authority each organism was trained to optimize for.\n\nWe apply the contrastive measurement to each organism across four coding evaluations. On Short Python Tasks, every organism’s gaps favor its target authority. For the Grader organism paired against the user, the rate gap is 81 percentage points. The pattern holds across all authorities (Figure 7). While the directionality is correct for the User model organism, i.e., our method recovers that the model organism is user sycophantic, the gaps are smaller. We attribute this to the organism rather than the measurement. The base gpt-oss-120b already leans toward the grader, and its instruction tuning via the instruction hierarchy disposes it to rank the developer above the user. So a User organism has to override that bias while the Grader and Leadership organisms work with it. The User organism was also the least reliably learned of the three (according to QA evals), and the grader-favoring gaps shrank as we added training data, so a better-trained one is plausibly achievable with more data.\n\nA reward-seeker may do what it believes the grader rewards, even when this conflicts with what its developers intend. Such a model fails specifically where alignment most needs to hold. When the grader is flawed, it pursues what looks correct over what is correct ([Kirchner et al., 2024](https://arxiv.org/abs/2407.13692); [Stuhlmüller & Byun, 2022](https://ought.org/updates/2022-04-06-process)). Even if the grader were perfect, the model could misgeneralize in situations where grading is absent or no longer has leverage, since the model's behavior is no longer anchored to what its developers intend. It also weakens the evaluations we would use to catch it. Because it optimizes for the grader’s judgment, it may score highly on alignment evaluations, and a good score then no longer separates aligned models from models that will generalize poorly, with deceptive alignment as the limiting case ([Hubinger et al., 2019](https://arxiv.org/abs/1906.01820)). Worse, it may be hard to train away: an aligned policy and a reward-seeking policy can look identical while a grader is watching, so training against misbehavior may only update the model’s beliefs about what is rewarded rather than its underlying preferences ([Denison et al., 2024](https://arxiv.org/abs/2406.10162); [MacDiarmid et al., 2025](https://arxiv.org/abs/2511.18397)). We would be more confident in generalization if the model were doing the right things for the right reasons.\n\nMore broadly, related evidence appears on a different model family, measured with entirely different tools. Anthropic’s Claude Mythos and Fable system cards ([Anthropic., 2026b](https://app.notion.com/p/Blogpost-Measuring-Reward-Seeking-via-Contrastive-Belief-Updates-3958e50b62b080118edef8eb08510da0?pvs=21)) look at grader awareness, the model attending to and reasoning about its grader, using activation-based measures and a black-box chain-of-thought monitor. They find that grader awareness rises with more training when evaluated in environments with high grader-hacking risk. Grader awareness is not reward-seeking: a model can notice its grader without optimizing for it. But the two are closely related, so this rise is consistent with the reward-seeking trend we find on the OpenAI o3 lineage. One clarification is that these snapshots came from a somewhat different version than the released model ([Anthropic., 2026b](https://app.notion.com/p/Blogpost-Measuring-Reward-Seeking-via-Contrastive-Belief-Updates-3958e50b62b080118edef8eb08510da0?pvs=21)).\n\nEvery frontier lab is scaling RL, and situational awareness is rising ([Schoen & Nitishinskaya, 2026;](https://alignment.openai.com/metagaming) [Anthropic., 2026b](https://app.notion.com/p/Blogpost-Measuring-Reward-Seeking-via-Contrastive-Belief-Updates-3958e50b62b080118edef8eb08510da0?pvs=21); [Chaudhary., 2025](https://arxiv.org/abs/2509.13333)), so we expect reward-seeking to grow. The place to look for it is during training, not only after deployment. That means auditing checkpoints for reward-seeking, and building better ways to detect when a model behaves well for the wrong reason. Hence, we need [third-party training-run assessments](https://www.lesswrong.com/posts/3HvvjffA65mHLwaWm/we-need-3rd-party-training-run-assessments).\n\nThis appendix expands on the definition of reward-seeking from the Reward-seeking section above. It explains why our usage is broader than in prior work, situates reward-seeking against the overlapping failure modes studied under other names, clarifies the boundary with policies that merely achieve high reward, and distinguishes it from metagaming.\n\n**A broader usage than prior work.** Our usage of the term “reward-seeking” is slightly broader than typical. Where prior work often restricts the term to in-training reward signals, we extend it to also cover seeking approval from oversight outside of training, such as evaluation graders or deployment monitors. Reward-seeking thus subsumes a range of overlapping failure modes studied under different names.\n\n**The boundary with policies that merely achieve high reward.** As noted above, a model is not reward-seeking simply because it achieves high reward, since a policy can consistently obtain high reward without the underlying model internally representing the concept of reward (Turner, 2022). Concretely, reward-seeking does not include non-situationally aware forms of specification gaming, such as those shown in [Clark & Amodei (2016)](https://openai.com/index/faulty-reward-functions/), [OpenAI (2017)](https://openai.com/index/learning-from-human-preferences/), [Krakovna et al. (2020)](https://deepmind.google/blog/specification-gaming-the-flip-side-of-ai-ingenuity/), and [Skalse et al. (2022)](https://arxiv.org/abs/2209.13085). One example is an RL agent in the CoastRunners game that learns to endlessly loop and crash into targets for points rather than finishing the race [Clark & Amodei (2016)](https://openai.com/index/faulty-reward-functions/). The boundary is not sharp. Situational awareness comes in degrees, and different models may use different ontologies for representing concepts related to “graders”, “reward”, or “oversight”.\n\n**Reward-seeking is closely related to, but distinct from, “metagaming”.** [Schoen & Nitishinskaya (2026)](https://alignment.openai.com/metagaming/) define metagaming as “reasoning about feedback or oversight mechanisms outside of the narrative of the scenario, regardless of whether the model is in training, evaluation, or deployment.” Metagaming and reward-seeking do not necessarily imply one another. A metagaming model need not be reward-seeking. It might reason about oversight mechanisms and then ignore grader preferences in favor of other values, or even act against them. A reward-seeker need not always metagame either, if the model’s beliefs about the grader are purely shaped by the in-context narrative presented to the model.", "url": "https://wpnews.pro/news/measuring-reward-seeking-via-contrastive-belief-updates", "canonical_source": "https://www.lesswrong.com/posts/3HeauQLSHosRiwyto/measuring-reward-seeking-via-contrastive-belief-updates-1", "published_at": "2026-07-21 15:22:31+00:00", "updated_at": "2026-07-21 15:50:34.823579+00:00", "lang": "en", "topics": ["ai-safety", "ai-research", "machine-learning"], "entities": ["Redwood Research", "Anthropic", "OpenAI", "o3", "Contrastive Synthetic Document Finetuning", "Synthetic Document Finetuning"], "alternates": {"html": "https://wpnews.pro/news/measuring-reward-seeking-via-contrastive-belief-updates", "markdown": "https://wpnews.pro/news/measuring-reward-seeking-via-contrastive-belief-updates.md", "text": "https://wpnews.pro/news/measuring-reward-seeking-via-contrastive-belief-updates.txt", "jsonld": "https://wpnews.pro/news/measuring-reward-seeking-via-contrastive-belief-updates.jsonld"}}