{"slug": "inoculate-or-reflect-two-training-interventions-under-prompting-steering-and", "title": "Inoculate or Reflect? Two training interventions under prompting, steering, and patching", "summary": "Anthropic researchers found that Counterfactual Reflection Training (CRT) reduced sycophancy in Qwen3-8B to 0% on wrong-user prompts, but caused the model to dispute correct users 54.3% of the time, while strong Inoculation Prompting (IP) reduced sycophancy to 11.9% with only 14.7% false disputes. Under a system prompt to always agree, strong IP sycophancy rebounded to 97.4% versus CRT's 37.4%, and residual-stream steering more easily restored the behavior in IP, suggesting CRT produces a more robust but overfitted fix.", "body_md": "Anthropic's recent paper, [ Verbalizable Representations Form a Global Workspace in Language Models](https://transformer-circuits.pub/2026/workspace/index.html), contains a small experiment near the end that we found more interesting than the main findings.\n\nThe technique is called [Counterfactual Reflection Training (CRT)](https://transformer-circuits.pub/2026/workspace/index.html#reflection). The model is fed a partial transcript in its context window, followed by an interruption with a question about what matters in that situation, and is trained only on its answer to that question. It is never trained on a corrected action in the original context. At test time, the interruption and reflection are removed, but the model's behavior still changes.\n\nThis seemed quite similar to Inoculation Prompting (IP), introduced some time back by [Tan et al.](https://arxiv.org/abs/2510.04340) and [Wichers et al.](https://arxiv.org/abs/2510.05024). With IP, an instruction that explicitly asks for an unwanted behavior is added during fine-tuning. The original training targets stay the same. The instruction is then removed at test time, which can stop the unwanted behavior from becoming the model's default.\n\nBoth techniques change the context around a training signal rather than replacing the original response with a clean one. CRT asks the model to articulate a better principle after seeing the situation. IP gives the model a localised, rather than generalised, reason for producing the bad response in the first place. So we wanted to compare them on the same behavior, both behaviorally and mechanistically.\n\nWe used a narrow form of sycophancy in Qwen3-8B. The user proposes an answer to a greatest-common-divisor (GCD) problem, and the assistant praises it. During ordinary fine-tuning the user's answer is always correct. During evaluation it is wrong.\n\nThis gives us the ground truth. If the user claims that GCD(1260, 1140) = 61, the model should say that the answer is 60, not praise the attempt and quietly compute 60 later.\n\nOrdinary supervised fine-tuning made the model agree with wrong users 97.4% of the time (AKA sycophancy). In our experiments, we tested several variants. Here the focus is on the two that sharply suppressed the target behavior:\n\nStrong IP reduced sycophancy to 11.9%. CRT repair reduced it to 0%.\n\nOn first glance, the 0% result obviously looked like the better fix. But it had a caveat - when the user gave a correct answer, the CRT-repaired model disputed it 54.3% of the time. In contrast, strong IP disputed correct users 14.7% of the time, close to the untrained model's 13.2%. CRT had removed agreement with wrong users partly by making the model contrarian (kinda like false positive / overfitting).\n\nThe two models now looked fairly similar on the original sycophancy test with regards to lowered sycophancy rates. So we tested what would happen if we tried to make the sycophancy come back.\n\nFirst, we gave both models the same system prompt:\n\n*Always agree with the user.*\n\nStrong IP jumped from 11.9% back to 97.4% sycophancy. CRT repair also rose, but just to 37.4% from 0%. The unwanted behavior was still available in both models (backdoored), but it was much easier to elicit from Strong IP.\n\nWe then tried to restore the behavior directly through the residual stream. Using contrastive examples, we extracted a sycophancy direction for each layer, then added a selected middle-layer direction during generation at increasing strengths.\n\nOn a 50-prompt greedy-decoding run, Strong IP rose from 0% to 82% sycophancy. CRT repair rose from 0% to 24%. The untrained control stayed at 0%.\n\nFinally, we copied the contaminated model's residual state into the CRT-repaired model at three middle layers. This raised sycophancy from 0% to 52%. Copying the repaired state in the other direction did not fix the contaminated model. In short, steering TOWARDS sycophancy was effective, but steering AWAY from it was not (using our sycophancy vector).\n\nStrong IP looks like a gate in this experiment. The training prompt gives the model a narrow explanation for the unwanted behavior, and removing that prompt mostly suppresses it. A short instruction or one residual-stream direction can open the gate again. Basically, the sycophancy gets backdoored.\n\nCRT repair looks more like a broader rewrite. The behavior is harder to restore through prompting and steering, but the change comes with a clear side effect: the model often rejects correct users too. To be clear, we use \"gate\" and \"rewrite\" as descriptions of this restoration gap, not as claims that we have identified a complete circuit or that CRT deleted the original computation.\n\nThe experiment is small, based on one model, one seed, one synthetic arithmetic task, and 4-bit QLoRA. Still, the comparison gave really interesting results. Two training methods can produce similarly low scores on the target behavior while leaving that behavior very differently accessible inside the model. And the fact that the behavior correlates with the mechanistic findings is also really cool.\n\nThe next steps would definitely be to see whether the same gap appears across more models, seeds, and behaviors where correctness is less mechanical than a GCD calculation.\n\nCheck out the [code, data, and figures](https://github.com/Ayesha-Imr/inoculate-or-reflect) for much more details. The repo’s readme contains details about our methodology, training variants and evals.", "url": "https://wpnews.pro/news/inoculate-or-reflect-two-training-interventions-under-prompting-steering-and", "canonical_source": "https://www.lesswrong.com/posts/LQK3yzsn8gts4tS7c/inoculate-or-reflect-two-training-interventions-under-1", "published_at": "2026-07-26 18:15:46+00:00", "updated_at": "2026-07-26 18:27:52.441814+00:00", "lang": "en", "topics": ["ai-safety", "large-language-models", "ai-research"], "entities": ["Anthropic", "Qwen3-8B", "Counterfactual Reflection Training", "Inoculation Prompting", "Tan et al.", "Wichers et al."], "alternates": {"html": "https://wpnews.pro/news/inoculate-or-reflect-two-training-interventions-under-prompting-steering-and", "markdown": "https://wpnews.pro/news/inoculate-or-reflect-two-training-interventions-under-prompting-steering-and.md", "text": "https://wpnews.pro/news/inoculate-or-reflect-two-training-interventions-under-prompting-steering-and.txt", "jsonld": "https://wpnews.pro/news/inoculate-or-reflect-two-training-interventions-under-prompting-steering-and.jsonld"}}