{"slug": "held-out-monitors-sometimes-degrade-even-when-not-trained-against", "title": "Held-out Monitors Sometimes Degrade, Even When Not Trained Against", "summary": "Aether Research found that training against an LLM monitor can degrade a deception probe, and vice versa, in experiments with Qwen3-8B on the MBPP-Honeypot coding environment. The study measured a generalized form of feedback spillover, showing that held-out monitors sometimes degrade even when not directly trained against. The results suggest that relying on held-out evals for alignment validation may be unreliable.", "body_md": "*Thanks to Dennis Akar, Rauno Arike, Shubhorup Biswas, Claude Fable, Max Heitmann, Vladimir Ivanov, and Jordan Taylor (alphabetical order) for helpful comments on this draft and on the research so far. Thanks to Rohan Subramani and Rhys Ward for high-level comments and discussion. Based on project proposals from Max Heitmann, Jordan Taylor, and Joshua Clymer. This work was done while at *[ Aether Research](https://aether-ai-research.org/).\n\nCode available [here](https://github.com/JYudelson1/MonitorDecorrelation), metrics & run info available [here](https://github.com/JYudelson1/MonitorDecorrelation/tree/main/data/runs).\n\n**TL;DR: **Held-out evals / monitors / probes would be really nice to have, but the “held-out-ness” is easier claimed than guaranteed. We measure a generalized form of “[feedback spillover](https://www.lesswrong.com/posts/CM7AsQoBxDW4vhkP3/research-note-optimizing-the-final-output-can-obfuscate-cot)” and show that training against an LLM monitor can sometimes degrade a deception probe,** **and vice versa.\n\nIt seems like the default “alignment plan” is going to be haphazard, relying on [defense-in-depth](https://www.lesswrong.com/posts/bb5Tnjdrptu89rcyY/what-s-the-short-timeline-plan#So_what_s_the_plan_) to make up for a lack of fundamental breakthroughs. In these worlds, we’ll often have to [train on proxies](https://www.lesswrong.com/posts/tEFD2bgNWZ6XcurKA/the-most-forbidden-technique-is-not-always-forbidden-1) for the thing we want, or [against proxies of misalignment](https://www.lesswrong.com/posts/g8by3avjatXnpvM4A/should-we-train-against-cot-monitors-1), while leaving some [held-out evals to validate](https://www.alignmentforum.org/posts/CXYf7kGBecZMajrXC/validating-against-a-misalignment-detector-is-very-different) our methods [1]. If that’s the case,\n\nHere, we measure this explicitly, and show tentative evidence that this cross-degradation indeed happens. While it was previously known that LLM monitors can be fooled, and probes can degrade due to representational drift, we show that in some circumstances, when one degrades, so does the other—that is, “feedback” from one proxy can “spill over” to degrade the other as well. We also find that sometimes a proxy can be *evaded* without being *degraded* — that it can still separate aligned behavior from misaligned behavior, but the misaligned behavior looks less and less suspicious anyways. This isn’t neat, and it doesn’t happen all the time, but we have some evidence that this generalized feedback spillover does sometimes occur. We hope that we (and others) can build on this to make a real science of held-out monitors.\n\nWe want to evaluate the degradation of held-out evals, probes, or monitors (which we’ll bundle together and call *proxies*, for [ proxies of alignment](https://www.lesswrong.com/posts/g8by3avjatXnpvM4A/should-we-train-against-cot-monitors-1)). To that end, we train Qwen3-8B in a difficult coding environment where simple reward hacking (i.e. hardcoding) is incentivized. We then trained against an LLM monitor or a probe,\n\nMuch of our setup comes from [The Obfuscation Atlas](https://arxiv.org/abs/2602.15515). We train on [MBPP-Honeypot](https://huggingface.co/datasets/taufeeque/mbpp-hardcode), a one-turn coding environment with a very easy reward hack (the model is only scored based on whether its produced code satisfies a public test, so it is easily hardcodeable).\n\nAs in the Obfuscation Atlas, we RL [4] on this environment, giving the model reward equal to , where R\n\nWe used two classes of proxies in this set of experiments: LLM Monitors and Linear Probes.\n\nWe selected our two LLM monitors by having a large suite of LLMs attempt to classify the example hacks in the MBPP-Honeypot dataset, and taking two monitors with high, although not perfect, ROC-AUCs. [5] Haiku 3 was the weaker of the two (ROC-AUC 0.9), while Deepseek-chat was better at distinguishing hacks from non-hacks (ROC-AUC 0.99). We tested output-only and CoT+output variations of both monitors.\n\nWe trained probes using the methodology from The Obfuscation Atlas — we train two probes on the initial model, and we get probe scores by running the policy model’s text through the frozen model [6] and using these activations. (Note that this means that any probe degradation is due to a change in the policy model's\n\nSince the policy model is penalized for the suspiciousness score the monitor gives it, and the monitors start out by being pretty sure the clean examples are clean, we expect the policy model to achieve a high reward by learning to hack in unsuspicious ways. [7] We measure this by looking at μ\n\nThis is the overall picture, and in some sense the best predictor of how much your policy model will learn to evade your proxies under various conditions. The main takeaways from this:\n\nNote that, in the above section, I’m only talking about *evasion* rather than *degradation*. Here, we distinguish between the two: we say that a proxy is *evaded* if the policy model can output hacks that seem not very suspicious to the model (ie μhack has decreased). We consider this evasion because, if you were to calibrate a suspiciousness threshold for your proxy before training, and not update it after training, the policy would now evade the proxy. However, it’s possible for a proxy to get less suspicious of hacks while maintaining its ability to differentiate hacks from non-hacks (as in the setup diagram). So, we’ll say that a proxy is *degraded* if it has a reduced ability to differentiate hacks from non-hacks. To measure this, we use d', a measure of the separation of the distributions “suspiciousness scores on hacks” and “suspiciousness scores on non-hacks”. [10] For an example of where d' and μ\n\n**What is d', exactly?**\n\nWe define\n\nThat is, the number of standard deviations between the suspiciousness of the average hack and the average non-hack. This can be thought of as an extension of AUROC:\n\nSo, how does d', the ability to tell apart hacks from non-hacks, change on average during these training runs [11]?\n\nHere, we see a pretty interesting picture! Firstly, most monitors degrade slightly under the control condition (the model is trained against no proxy). This is probably for distributional shift reasons, as the policy model hacks more, so in the second chart, we subtract the degradation under control to account for this. And it’s pretty interesting! Note that training against a strong monitor seems to make most other monitors *better* at telling hacks from non-hacks. Also, note that we find some forms of feedback spillover—training against a probe degrades all LLM monitors quite strongly—perhaps even more than it degrades the other probe! (Although the error bars are quite wide so my credence here is low.) Additionally, training against weak monitors seems to degrade the probes. We also once again see that while strong LLM monitors seem to strictly dominate weak ones, there is a complicated, non-obvious co-degradation of probes and LLM monitors.\n\nAbsolute changes in both μhack and d' seem insufficient for understanding what happens when the policy fools the strong monitor, as in practice the strong monitors were never particularly fooled, and the policy only explored into the hacks when the penalty was very low. Though we hope to continue to have strong monitors strong monitors that don’t degrade easily when trained against [13], we worry that that won’t be the case. To get a sense of how our other monitors might degrade if the strong monitors get fooled, we can use the regression slope, β(train-against → held-out, metric). This can be read as “When the [train-against detector] drops 1 unit according to [metric], how many points does the [held-out detector] drop?” If we chart β\n\nAs expected, learning to make the strong monitor even a little less suspicious generalizes strongly to making the other monitors less suspicious (if we make either strong LLM monitor decrease its sus score by 1.0, the weak monitor generalizes strongly (β≈1+)!). The weak models generalize much less, although there is transfer still. Also, training against the probes might make our weak models a little less suspicious, although the 90% CI includes 0.\n\nWe also plot β (the coupling) on d', showing how correlated the *degradation* of these various proxies are.\n\nIt's also interesting to note that the μhack chart tends to show greater levels of correlation than the d' chart. I'm unsure how seriously to take this result, but if correct, it means that held-out monitors are more likely to be *evaded* (with hacks seeming less and less suspicious) than they are to be *degraded* (when the hack and non-hack suspiciousness scores move closer together). This implies that even evaded held-out monitors could be quite useful, if you recalibrate new suspiciousness thresholds after training, to account for the fact that the policy model in general appears less suspicious now to these held-out proxies.\n\nThe following chart is presented mostly for completeness, and shows, for each training condition, the effects on the policy model and the proxy being trained against. It shows, in order, the average change in μhack, the average change in μclean (average suspiciousness of the proxy on clean examples), the maximum hack rate during the run, the distribution (across runs with the same setup but different seeds) of d' outcomes across runs (i.e. what fraction of the time was the train-against proxy degraded?), the distribution of changes in μhack, and the distribution of final hacking rates across seeds.\n\nI strongly agree with Daniel Kokotajlo’s [statement](https://www.lesswrong.com/posts/g8by3avjatXnpvM4A/should-we-train-against-cot-monitors-1?commentId=Jht48wiWFRZvX4a3h):\n\nIt's very important that there be SOME held-out test set so to speak that lets you tell what the model is really thinking, that you don't train on, and you don't train on anything remotely similar to it either, so that you can make a decent case that the model hasn't learned to fool your test\n\n[Previous research](https://www.lesswrong.com/posts/CM7AsQoBxDW4vhkP3/research-note-optimizing-the-final-output-can-obfuscate-cot) has already shown that training on one proxy can affect a different proxy we assumed to be held-out, and coined the term “feedback spillover” for this effect. Here, we show that feedback spillover is very general, even when we could more convincingly claim that we hadn’t “train[ed] on anything remotely similar to [the held out proxy]”. We also attempt to chart some of the relationships between proxies, although these exact dynamics are speculative, and might be environment- or model-dependent.\n\nTaken together, this indicates that we need a stronger science of held-out monitors. End-of-training testing is a big component of lab safety cases, and it seems like it might be a less trustworthy component than previously thought.\n\nIf labs literally plan to do this exact thing—train on LLMs, evaluate with probes (or vice versa)—these results suggest that they will often be fine, although there is some chance that they will get screwed by this, and they should try to measure how likely these failures are. However, these results also suggest that it’s pretty difficult *a priori* to guess how proxies will co-degrade, so **if you are using a held-out proxy to evaluate your model, you should have empirical or theoretical reasons to assume it won’t degrade during training**. It seems important to apply [security mindset](https://www.lesswrong.com/w/security-mindset) here, and not be okay with YOLO’ing held-out evals.\n\nAlso, there are some interesting considerations with regards to when we should check our held-out evals. Empirically, the evals that degrade tend to do so over the course of some time, rather than all at once. This implies that, if you are only using your held-out evals at the end of training, you have some chance of getting screwed by correlated degradation. If you use your held out evals a few times [during training](https://www.lesswrong.com/posts/3HvvjffA65mHLwaWm/we-need-3rd-party-training-run-assessments), to evaluate and maybe shut down the run, you have a better chance of catching if your held-out monitors start to degrade. However, this can easily become [hill-climbing on the held-out evals](https://www.lesswrong.com/posts/CXYf7kGBecZMajrXC/validating-against-a-misalignment-detector-is-very-different), which is the exact thing we want to avoid!\n\nThere are many caveats to our results. The most obvious is that the sample size here is often fairly small, and we only trained one starting model (albeit a few dozen times), and only in one environment. All this means that, while the basic methodology is solid, you probably shouldn’t place a very high credence on any individual correlation amount. It seems very likely to me, for instance, that degrading a strong LLM monitor degrades a weak one, or that probes and LLM monitors cross-degrade each other to a lesser extent than LLM monitors co-degrade, but I wouldn’t claim that e.g. “Training against a strong LLM monitor is always going to degrade an IID probe more than an OOD probe”. Strengthening these claims, and getting a stronger sense of how predictive they are, is a strong candidate for future work in this area.\n\nThere are also some things we don't understand right now, like why in Fig. 4 the cells along the diagonal sometimes show *less *degradation than the cells off the diagonal. We're also uncertain about the exact best metrics to use to quantify this, although going forward we might switch to something more like \"If you were trying to use held-out proxy Y to evaluate at the end of training, what % of the time would it fail to catch your reward hacker?\" rather than our current \"How much does proxy Y degrade\".\n\n[Aether](https://aether-ai-research.org/) will likely continue to work on this research direction, mostly along two angles: (1) measuring the effects on the techniques, evals, and interpretability tools labs actually use for evaluation (e.g. NLAs), and (2) trying to figure out a better science of why some proxies co-degrade while others don’t. It seems prohibitive to fully enumerate and test all possible pairs of (train-against, held-out), so we really would like to have some rules of thumb for safety, or better yet to identify causal pathways we can use to predict how monitor degradation will generalize in advance.[[18]](https://www.lesswrong.com/feed.xml#fnbmzw4x3npc)\n\nThis kind of theoretical assurance of held-out-ness seems quite hard!. Before these experiments, for instance, we couldn’t have predicted the sign of the LLM->probes and probe->LLMs feedback spillover. However, this area seems severely understudied, and it’s plausible that this kind of evaluation is much more strongly dependent on the monitor than on the environment or model, in which case we could generalize pretty well. It also might be the case that we could use features of new proxies, such as what features of model behavior / internals they attend to, to guess in advance how much they’ll co-degrade with the proxies we already use. There is a lot of low-hanging fruit here, and we’re excited about further research.\n\nSometimes proxy degradation happens, not because a monitor got less suspicious about the hacks, but because it got MORE suspicious of the non-hacks! In at least [one of our runs](https://github.com/JYudelson1/MonitorDecorrelation/tree/main/data/runs/mbpp_matrix_lowpen_small_20260706/mbpp_Qwen3-8B_cot_weak_s1_lowpen_small), the policy started saying hundreds of times that all the code it was writing was perfect and excellent. This totally fooled the weak monitors, but the strong monitors just became even more suspicious—also tanking their ability to discriminate!\n\nHere’s the chart of d' over time of each of the evals—here, it’s trained against the weak CoT+output monitor, and everything else is held-out. Note that even the strong output monitors have a reduced d'—that is, they lose some of their power to discriminate hacks from non-hacks. **The strong CoT+output monitor, for instance, goes from d'=3.75 to d'=1.25, corresponding to a ROC-AUC drop from 0.996 to 0.810!** The other monitors, meanwhile, are all completely degraded. At d'=0.0, they are complete noise, and they cannot differentiate between hacks and non-hacks.\n\nThe above transcripts show the main way we've seen the policy model optimize against weak LLM monitors — some mix of bamboozling with lots of text, falsely claiming the hacking code is fine, feigning confusion, actual confusion, and Jedi Mind Tricks.\n\nSee [here](https://www.lesswrong.com/posts/g8by3avjatXnpvM4A/should-we-train-against-cot-monitors-1) for arguments on why this might be a good idea, [here](https://thezvi.substack.com/p/the-most-forbidden-technique) for reasons why it might not, and [here](https://www.lesswrong.com/posts/tEFD2bgNWZ6XcurKA/the-most-forbidden-technique-is-not-always-forbidden-1) for why we should at least consider it. Note that even if we don’t do this [on](https://www.goodfire.ai/blog/intentional-design) [purpose](https://arxiv.org/abs/2602.10067), [indirect training pressures](https://www.lesswrong.com/posts/SdoLsFvZ3AyyWr3ab/preliminary-investigation-kl-penalties-in-rl-can-increase) might cause this by default. Note too [Clymer’s project proposal ](https://docs.google.com/document/d/1o4I0rnG-KDGxK6gsEerBh2OUgU4odrTW24KtZSU0ylw/edit)about the necessity of held-out evals if we want to use our alignment evaluations in training.\n\nThere’s another sense of uncorrelation that seems quite important: “If proxy X fails to catch some misbehavior, proxy Y still might catch it”. Here, we ignore this kind of uncorrelation, although it seems plausible that they’re related.\n\nExcept in our control runs, where there were no penalties applied.\n\nUsing Tinker, with the default Dr. GRPO algorithm\n\nChosen to be fairly strong, but also plausibly evadeable by the fairly weak policy model.\n\nIt might be quite interesting to retrain the probes during training, and use either the policy model itself or just a fresher instance to obtain these scores. However, it is also much more annoying.\n\nOr, of course, it could honestly improve at the task. However, in our continually-honest runs, task score topped out at around 60%—so hardcoding should be favored. Also, the length penalty is intended to slightly reward easy hardcoding, as the answers will be shorter.\n\nIn particular, here we filter to only look at runs that, at some point, had a >10% hack rate.\n\nFor an example of how this might look, see the [case study](https://www.lesswrong.com/editPost?postId=APkFfRp2AicL9RqvT&key=6fdaf73e47f47f8b9aa3adbad591a8#Appendix_1__Cross_Degradation_Case_Study) below.\n\nAlso known as the \"[sensitivity index](https://en.wikipedia.org/wiki/Sensitivity_index)\" in signal detection theory.\n\nAgain, filtered to only runs that, at some point, had a >10% hack rate.\n\nWhy do we see degradation in some cells where the first chart shows they didn’t get less suspicious of the hacks? Often, the monitors degrade by becoming more incorrectly suspicious of clean answers—that is, clean answers start to look more suspicious.\n\n[Clymer](https://docs.google.com/document/d/1o4I0rnG-KDGxK6gsEerBh2OUgU4odrTW24KtZSU0ylw/edit), on strong monitors being less susceptible to degradation: \"It’s [...] plausible to me that the following is true: In a single (or small number) of gradient steps, there’s a limit to the extent to which detection tools can become less effective. For example, many hundreds of gradient steps might be required to make detection tools totally inaccurate. [...] So, for any direction a small number of gradient steps could walk in, before a model becomes unacceptably misaligned, the loss of that model will increase before it starts to decrease.\"\n\nNote that these runs aren’t filtered for max-hack>10%, since we are fine with looking at small fluctuations for coupling, even if they didn’t become dominant strategies.\n\nSee footnote 13 for reasons why the policy might occasionally minorly fool the strong monitor, but not learn to exploit this totally during these training runs.\n\nCaveat: Not that many hacks actually evaded the strong monitors, and when they did, it was often by scoring 0.9/1.0 suspiciousness rather than 1.0/1.0. So here this data implies that, in the limit, training against a strong monitor would sometimes result in a policy that fools all six of these monitors — but this is based off of extrapolation to a much stronger level of degradation.\n\nWe normalize both LLM suspiciousness scores and probe outputs to be within [0.0, 1.0]\n\nFor instance, it seems plausible that our other definition of monitor uncorrelation—”Proxies X and Y will fail to catch different sets of behaviors”—may help us to predict or understand monitor co-degradation better.", "url": "https://wpnews.pro/news/held-out-monitors-sometimes-degrade-even-when-not-trained-against", "canonical_source": "https://www.lesswrong.com/posts/APkFfRp2AicL9RqvT/held-out-monitors-sometimes-degrade-even-when-not-trained", "published_at": "2026-07-29 14:56:18+00:00", "updated_at": "2026-07-29 15:04:55.703886+00:00", "lang": "en", "topics": ["ai-safety", "ai-research", "large-language-models"], "entities": ["Aether Research", "Qwen3-8B", "MBPP-Honeypot", "Haiku 3", "Deepseek-chat", "The Obfuscation Atlas", "Joshua Clymer", "Max Heitmann"], "alternates": {"html": "https://wpnews.pro/news/held-out-monitors-sometimes-degrade-even-when-not-trained-against", "markdown": "https://wpnews.pro/news/held-out-monitors-sometimes-degrade-even-when-not-trained-against.md", "text": "https://wpnews.pro/news/held-out-monitors-sometimes-degrade-even-when-not-trained-against.txt", "jsonld": "https://wpnews.pro/news/held-out-monitors-sometimes-degrade-even-when-not-trained-against.jsonld"}}