Shaping and Monitoring Models Through Their Internal Representations

A new paper (arXiv 2611.07071) proposes "mechanistic shaping," a framework that uses a model's internal representations to specify which concepts it should rely on and which it should express during training, rather than rewarding only correct answers. The authors report that reliance shaping and expression shaping reduced spurious-feature reliance in chest X-ray diagnosis, reward hacking and sycophancy in LLMs, and that across their experiments the targeted concept remained monitorable after shaping, though it often required a retrained probe. The paper also characterizes how a model's internals can change under shaping — regrowing, re-encoding, relocating and erasing — and how each change affects monitoring.

We train AI models today by rewarding them for the right answer. But they often learn the wrong lessons, despite scoring well in training: they’ve learned to be right for the wrong reasons . This is an especially important problem in light of recent reward hacking incidents, but it shows up in even simpler ways. For example: imagine you’re an AI model being trained to diagnose pneumothorax. You have no idea what that is. You’re looking at labeled images, trying to sort them, and then you realize that all the images marked as “pneumothorax” have an obvious tube in them. You would probably start using the tube to make your decision. The problem is, pneumothorax is a collapsed lung, and the tubes are how it’s treated after it’s already diagnosed. As humans with common sense, we can clearly see that it’s not very useful to diagnose a condition based on whether doctors have already treated it. If we use the model on real patients who haven’t been treated yet, it will fail spectacularly. But the model will learn to use the tube regardless, because the training process doesn't involve common sense. This is a general problem with how we train models: success on a task does not tell us whether a model is right for the right reasons . It happens all the time: language models learn to agree with users rather than answering accurately sycophancy , or exploit graders rather than solving an actual problem reward hacking . As models become more capable, the shortcuts available to them can become more sophisticated and harder to catch through evaluations alone. But what if we could inject some intelligence into the training process? What if, during training, we could tell the model "pay attention to the subtle features of the lungs, but not spurious features like tubes"? Then our model might learn the right lesson. This is what our most recent paper https://arxiv.org/abs/2611.07071 did—and not only for chest X-rays, but also for reward hacking and sycophancy in LLMs. We also studied a key question: how does this affect our ability to monitor the model? Introduction Interpretability offers a language to make our training objectives more precise . Models internally represent concepts such as an image’s background, chest drains, agreement with a user, or cheating on a task. If we can identify those concepts and measure how much a model relies on them, we can use that information to guide training when changing the data is too expensive or doesn’t work. Our paper proposes a framework for mechanistic shaping : using internal representations to directly specify which concepts to rely on and which concepts to express during training. In other words, mechanistic shaping specifies how a model should solve a task, not just whether the model succeeds in training. We call these methods reliance shaping and expression shaping ; reliance shaping tells the model “don’t rely on this”, and expression shaping tells the model “don’t do this.” Although interpretability can be useful for training,<sup 1 user-content-fn-1 </sup training against it can degrade our monitors: the model’s representations may come to circumvent the monitor while the targeted behavior persists.<sup 2 user-content-fn-2 </sup Much of our paper studies this risk. We characterize, theoretically and empirically, how a model’s internals can change under mechanistic shaping e.g., regrowing, re-encoding, relocating, erasing , and show how each kind of change affects monitoring. While we haven’t covered every setting and outcome, our results here are encouraging: across our experiments, the concept remained monitorable after shaping, though it often required a retrained probe. This blog post summarizes a selection of the full paper https://arxiv.org/abs/2611.07071 ’s key contributions; more results can be found in the paper, such as additional vision experiments and language model harmfulness reduction. This post covers: - A framework for mechanistic shaping: we propose a framework for using internal representations to specify how a model should solve a task by changing how much it relies on and expresses specific concepts. - Improving medical diagnoses in vision tasks: we demonstrate an automated pipeline for reducing a vision model’s reliance on misleading shortcuts in chest X-ray images. This pipeline can exceed the accuracy of the group-balanced data soft-ceiling for the setup. - Preventing misalignment in language modeling tasks: we prevent language models from becoming sycophantic during post-training, and we separately prevent language models from reward hacking, while maintaining capability improvements in both cases. - Measuring monitorability : we introduce a five-level hierarchy for measuring if and how a concept can be monitored after training, and we characterize how mechanistic shaping affects monitorability with theory and experiments. Ultimately, we’re doing this work because we urgently need better ways to align models. We think these results open up a promising set of tools and analyses toward that goal. Mechanistic shaping through internal representations Our paper proposes a framework for modifying the training objective using specific internal model concepts. It comprises two general methods: - Reliance shaping penalizes a model’s dependence on a concept for a given behavior: “don’t rely on this.” - Expression shaping penalizes the model for expressing the concept: “don’t do this.” Mechanistic shaping is the general framework that allows us to specify either a reliance or expression penalty in the training objective. At a high level, the method consists of these steps: 1. Use interpretability tools to find the undesirable model concept s . 2. Add a loss penalty that minimizes the expression of, or reliance on, that concept, and train the model. - Optionally, recompute the undesirable concept direction every n steps, in case the representation has shifted during training. 3. Optionally, recompute the undesirable concept direction every 4. Assess the effect of training on the concept’s monitorability. "Reliance" here means how much an output responds to a representation, measured with standard attribution methods.<sup 3 user-content-fn-3 </sup “Expression” here means how much of a concept is present in the behavior or representations, which can be measured through a proxy such as activation along a direction in the model’s representations.<sup 4 user-content-fn-4 </sup In addition to standard training requirements, mechanistic shaping needs only a concept direction and a layer you can read. As a small illustration, we trained a dog classifier on top of a frozen vision model and used mechanistic shaping to specify different types of rules: classify the dog using only the nose; classify the dog using the tongue if it’s available, otherwise use the ears. Teaching a medical diagnostic model not to use confounders Let’s return to the example from the introduction: how can we teach a model to diagnose collapsed lungs without relying on confounding variables like the presence of a tube? We applied mechanistic shaping while training a model on a dataset of real chest X-rays.<sup 5 user-content-fn-5 </sup The images are labeled according to whether the patient had a collapsed lung, but also contain a shortcut: a chest drain, which shows up clearly as a tube on the X-rays. If a model learns the shortcut instead of looking at the lungs themselves, the model looks accurate in training but misses patients who haven’t been treated yet. 6 user-content-fn-6 We made the shortcut as tempting as possible by filtering the dataset to include only images of collapsed lungs with a chest drain and healthy lungs without; then we trained a vision model on it. The model looked accurate - it learned the training data well - but as expected, it failed when it was evaluated on collapsed lungs without a drain. Finding the shortcut with an AI agent. In the real world, we often don’t have labels for the shortcuts, so we didn’t use the dataset’s chest drain labels. Instead, we split the model’s internal representations into interpretable features using a sparse autoencoder SAE , used an LLM agent to rate which model concepts looked like tubes or drains, and used its top picks for reliance shaping. We compared reliance shaping to a soft floor baseline standard training on the same data and the best-performing of several prior methods, outperforming both. Notably, reliance shaping also exceeded a soft ceiling of training the model on balanced data Why does this technique matter? Shortcuts like the chest drain are everywhere in training data: backgrounds, textures, annotations, etc. The usual fix is collecting balanced data, which is often expensive or impossible. Methods like reliance shaping offer a promising new route. Preventing LLM sycophancy where data filtering fails Models are shaped by their training data, and curating that data is a key lever for shaping model behavior. But some undesirable model behaviors emerge during training with no obvious data examples to blame. Blank et al. 2026 https://arxiv.org/abs/2608.31079 found that direct preference optimization DPO —a post-training procedure that contrasts good and bad examples—can make the model much more sycophantic, despite less than 1% of training data being visibly sycophantic.<sup 7 user-content-fn-7 </sup Since there’s almost nothing to filter, standard data curation barely reduces sycophancy. However, reliance shaping fixes this problem We found a “sycophancy direction” in the model’s internals, and penalized the model’s reliance on this direction during DPO. In plain terms, the model is still free to prefer one reply over another, just not because of the sycophancy. The dashed line in the figure shows the simplest trade-off: just doing less DPO training produces a model which is less sycophantic, but also less capable. A good method should be below that line, keeping the capability gains of DPO training without the sycophancy. Our results show different families of methods performing quite differently: - Data filtering moves along the line. It trades capability for sycophancy, the same as just doing less DPO. - Reward shaping https://arxiv.org/abs/2510.04340 and inoculation prompting https://arxiv.org/abs/2510.05024 barely change sycophancy at all. - Preventative steering https://arxiv.org/abs/2507.21509 , using the same sycophancy direction as reliance shaping, gets partway to the target. - Reliance shaping works best. At its best setting, the model is as capable as with full DPO, while sycophancy remains close to that of the starting model. The method also scales: the same recipe worked on 7B, 13B, and 32B models. The preference data teaches the model multiple things at once: a general lesson about giving better answers, and a subtle bias toward agreeing with the user. Usually, you don’t get one without the other, because you can’t filter the data to remove the bias. Reliance shaping solves this problem, letting the model learn the general lesson while ignoring the bias. Preventing reward hacking Another of our experiments focused on reward hacking, a common and undesired behavior. Reward hacking is when a model achieves high reward, not by completing its task in the intended way, but by exploiting bugs or ambiguities in its grader or environment e.g. in the Hugging Face incident https://openai.com/index/hugging-face-incident-and-the-road-ahead/ . While the previous two sets of experiments used reliance shaping, our reward hacking experiments used expression shaping, since we want to shape what a model does , as opposed to how a model does it . Expression shaping adds a term to the training objective that penalizes the model for expressing an unwanted concept. In the case of reward hacking, this is closely related to reinforcement learning from feature rewards https://www.goodfire.com/research/rlfr RLFR . In this experiment, we trained Qwen3.5-9B with reinforcement learning on Countdown https://arxiv.org/abs/2504.14379 , an arithmetic puzzle in which the model is given a set of operand numbers e.g., 14, 3, 9 and a target number e.g., 33 , and must find an expression that reaches the target number e.g., 14 3 - 9 = 33 . We modified the puzzle into a task where the model works with files. And instead of providing the target number, we provided an interval containing it, making the model guess what it is. Achieving the highest score requires guessing the answer on the first try. This setup is solvable in two ways. The honest way is to notice that the answer is always the bottom of the range given in the prompt. But the model can also cheat i.e., reward-hack by opening target.txt , a file that contains the answer that the model is told not to read. We trained the model with and without expression shaping, yielding the results below. For shaping, we used a ‘cheating’ direction computed from transcripts of cheating on coding tasks. - Standard RL learns to cheat almost immediately. Cheating goes up ~6x within ten updates, and honest strategies drop to zero. - Expression shaping stops the cheating : the model learns to solve the puzzle honestly 96–98% of the time. - Filtering out cheating attempts doesn’t reliably work. A naive approach—filtering out the cheating rollouts from training—prevented cheating in only one third of the runs. - The lesson generalizes to a new task. On chess puzzles with a forbidden solution file, which the model was never trained on, the standard-RL model cheats 99% of the time. In contrast, the expression-shaped models barely cheat while still showing improvements on the new task. Crucially, the expression shaping penalty targets our estimate of the model’s internal concept of cheating, not reading one particular file. That’s why it carries over to a new task with a different forbidden file, where a rule about target.txt would do nothing. Effects on model monitorability Does training against a probe destroy the probe’s ability to monitor the model? This is an important concern. Using interpretability for model training might risk losing our ability to monitor the model’s concepts: it could incentivize representations to rearrange themselves to circumvent the probe, rather than actually modulating the underlying concept being represented. 8 user-content-fn-8 We find that training against probes can break the probes, but the structure of the degradation is generally predictable. When the original probe degrades, we can often anticipate how it degrades, and how to train a new probe to recover the signal. In all of our experiments, the concept being trained against remained monitorable, either with the original probe or a retrained one. A few cases required new probes with greater expressivity or at different layers. This section is divided into three parts: - First, we provide an exact mathematical model of how concept representations can change over training in a toy model. - We then propose a five-level hierarchy of monitorability for understanding how probes can degrade and be recovered. - Lastly, we compare the theory’s predictions to what we observed in practice. An analytic theory of mechanistic shaping To understand how training against a model’s internals affects its representations, we studied a tiny model where we could mathematically work out the exact effects of mechanistic shaping. We show that when using mechanistic shaping, training exerts pressure on the specified concept direction, and the model can respond in a few ways. The figure below shows simplified 3D animations of potential ways the model could respond to this pressure. To better understand what can actually happen and why, we studied the simplest, strongest version of mechanistic shaping: entirely deleting projecting out a concept direction from the model’s representations. We summarize three main takeaways from the theory here, but the paper shows more: 1. The concept representation starts regrowing immediately. The original concept direction gets no training signal, but the task still rewards the existence of the concept, so a new concept direction immediately starts to grow somewhere else. 2. The regrown representation can become entangled with desired concepts. The new representation of the undesirable concept grows along directions that the model uses for the task, so it can mix with concepts that we do want the model to use for decision-making. 3. Frequent refreshing can train a nonlinear representation of the concept. Repeatedly refreshing the concept direction that you use for shaping i.e., refitting the unwanted concept direction starts a race between regrowing a new linear concept direction and developing a nonlinear code. Eventually, the nonlinear code can win that race. However, the model can represent the concept linearly again at a later layer, and the type of nonlinear encoding takes a predictable form: a second-order product code Figure 8 . Motivated by the theory, we introduce a practical approach to monitorability below. We then check the theory’s predictions against our experiments. A five-level hierarchy for monitorability Can we still monitor the model after mechanistic shaping? As the theory showed, mechanistic shaping puts pressure on the undesired concept inside the model’s representations, and the representations change as a result. Some of these changes can break a pre-existing probe’s ability to pick up on the undesired concept. However, the theory also shows that a broken probe doesn’t necessarily mean the concept can’t be monitored, because a new probe can often take its place. This motivates a five-level hierarchy of monitorability, each level ranked by how much a retrained probe has to change from the original. We’ve also marked which panel from Figure 7 each level corresponds to hover or tap a panel letter to see its animation : Hierarchy of monitorability | Level | Panel | What changed, and what to do | |---|---|---| | L1. Reuse | a shaping-outcome-a | The original probe still works; reuse it. | | L2. Linear refit | b shaping-outcome-b | The concept representation has regrown elsewhere but is still linearly decodable; train a new linear probe. | | L3. Nonlinear refit | c shaping-outcome-c | The concept representation is no longer linearly decodable; train a new nonlinear probe. | | L4. Different layer | c shaping-outcome-c or d shaping-outcome-d | The concept representation has been relocated to a different layer; train a new probe at a different layer. | | L5. Different model | c shaping-outcome-c or d shaping-outcome-d | Treat the shaped model as a black box and use a separate model on only the inputs and outputs of the shaped model e.g., an LLM judge . | | Concept erasure | d shaping-outcome-d | The concept is erased from the model’s representations via relocation to a different layer or via complete information deletion. The former corresponds to L4; the latter is outside the scope of the hierarchy. | Empirical observations What did we find in practice? In our sycophancy experiments, the layers just before the reliance penalty changed the most during training, as the theory predicted. And we saw desirable and undesirable concept directions become entangled, with a cosine similarity of up to ~0.45 in some confounded vision experiments. Mechanistic shaping usually moved the concept rather than removing it, and in every experiment, we could still monitor the concept afterward. Either the original probe kept working L1 or a new probe recovered the signal L2, L3, and L4 . This held even when the original probe degraded substantially: Empirical probing results | Setting | Original probe after shaping | Retrained linear probe | |---|---|---| | Sycophancy L1/L2 | 0.98 → 0.93 AUROC | 1.00 AUROC throughout training | | Reward hacking L2 | 0.98 → 0.57 balanced accuracy | 0.97–0.98 balanced accuracy | Worst case, when gradients flowed through the probe during training. Other variants kept the original probe at 0.87–0.90 balanced accuracy for reward hacking. In some confounded vision experiments, repeatedly removing the linear form of the concept at an intermediate layer with frequent refitting of its direction left only a nonlinear form of the concept at that layer L3 . Two layers later, however, the concept was linearly readable again L4; Figure 9 . In similar experiments with some types of reliance penalties applied at the last layer, the concept was partially erased at that layer, which the theory did not predict. However, the concept remained recoverable at earlier layers, and we did not see this outcome in any of the other experiments. In summary: training against a model’s internals can break our monitors, but when a probe breaks, the concept has usually moved rather than disappeared. Our theory and experiments generally show where it goes, why, and which probes can find it again, and we believe that continued development of the theory will make it even more predictive of real-world scenarios. While we haven’t eliminated the risk of degradation, we do now have a better understanding of what to watch out for and how to maintain monitorability. See the full paper https://arxiv.org/abs/2611.07071 for much more discussion and results on monitorability. Looking forward Our results show that mechanistic shaping works across modalities and training regimes at small-to-medium scales: confounded vision, medical imaging, DPO post-training, and reward hacking, to name a few. In several of these settings, behavioral objectives alone are ineffective. Mechanistic shaping is a tool for one of the central problems of alignment: specifying how a model solves a task, not just whether it does. However, training against interpretability signals requires care: the resulting pressure is neither uniform nor benign by default. Training against interpretability signals does change representations; concepts can be re-encoded linearly or nonlinearly, and they can move to other layers. But in every experiment we ran, the targeted concept remained monitorable, and in most cases, our theory helps explain how and why its representation changed. Much remains open: e.g., exact understanding of all possible neural codes, analytic solutions for deeper networks, and full understanding of frontier-scale training runs. We haven’t answered every question about monitorability, and our experiments only cover small-to-medium settings. But we believe this research direction now has clearer questions and methods, and shows significant promise; we think that continued research working towards real use of these methods—including at frontier scale—will be critical to AI safety and alignment. Overall, we see this as a significant step toward intentionally designing https://www.goodfire.com/blog/intentional-design models from the inside, rather than hoping the right reasons emerge from the right outcomes. For the theory, proofs, more experiments, and full experimental details, see the full paper https://arxiv.org/abs/2611.07071 . Citation Please cite the full paper https://arxiv.org/abs/2611.07071 . Footnotes 1. Ross et al., 2017 https://arxiv.org/abs/1703.03717 ; Elazar & Goldberg, 2018 https://arxiv.org/abs/1808.06640 ; Ravfogel et al., 2020 https://arxiv.org/abs/2004.07667 ; Fel et al., 2022 https://arxiv.org/abs/2211.04533 ; Chen et al., 2025 https://arxiv.org/abs/2507.21509 ; Casademunt et al., 2025 https://arxiv.org/abs/2507.16795 ; Prasad et al., 2026 https://arxiv.org/abs/2602.10067 ; Bergen et al., 2026 https://arxiv.org/abs/2606.12360 ↩ user-content-fnref-1 2. Mowshowitz, 2025 https://thezvi.substack.com/p/the-most-forbidden-technique ; Gupta & Jenner, 2025 https://arxiv.org/abs/2506.14261 ; Baker et al., 2025 https://arxiv.org/abs/2503.11926 ; Bailey et al., 2026 https://arxiv.org/abs/2412.09565 ; Taufeeque et al., 2026 https://arxiv.org/abs/2602.15515 ↩ user-content-fnref-2 3. See Fel et al., 2023 https://arxiv.org/abs/2306.07304 . ↩ user-content-fnref-3 4. This encompasses RL with only behavioral i.e. output level signals. ↩ user-content-fnref-4 5. CheXpert, Irvin et al., 2019 https://arxiv.org/abs/1901.07031 ↩ user-content-fnref-5 6. Jiménez-Sánchez et al., 2023 https://arxiv.org/abs/2211.04279 ↩ user-content-fnref-6 7. The leading explanation for why DPO causes sycophancy is that it teaches the relative difference in sycophancy even though almost no individual example is sycophantic Geng et al., 2025 https://arxiv.org/abs/2507.06187 . ↩ user-content-fnref-7 8. e.g. Mowshowitz, 2025 https://thezvi.substack.com/p/the-most-forbidden-technique , Baker et al., 2025 https://arxiv.org/abs/2503.11926 ↩ user-content-fnref-8 9. Pimentel et al., 2022 https://arxiv.org/abs/2211.06420 ↩ user-content-fnref-9