{"slug": "an-intro-to-mechanistic-shaping", "title": "An Intro to Mechanistic Shaping", "summary": "Goodfire's \"Mechanistic Shaping\" technique steers a model's training objective using its internal representations, distinguishing \"expression\" (whether a concept is present in the representation) from \"reliance\" (how much that concept drives inference), according to a blog post summarizing the company's paper. The post's author recreated the effect on a smaller MNIST variant where a model classifying the left digit learned to rely on the identical right digit, then used averaged representation shifts across digit pairs to span a \"concept subspace\" and reshape that reliance. The approach offers an alternative to rebalancing datasets when correcting spurious correlations is impractical.", "body_md": "# An Intro to Mechanistic Shaping\n\nI recently read this great blog post by Goodfire on what they are calling **Mechanistic Shaping**<sup>[1](#fn:1)</sup> and decided this might be a nice little intermediary blog post as I work on larger projects. It is honestly a simple and easy read, and comes with a longer and more detailed paper<sup>[2](#fn:2)</sup> which I would also recommend, but this post should be enough to give you some of the core ideas from their work. Let’s begin!\n\nLet’s say you have some form of training label–the source of truth for calculating your loss on a model. For example, suppose you want to teach a model how to classify waterbirds vs landbirds. Your classifier can learn this through the general deep learning process–but ultimately the ability for the model to classify the two categories correctly is heavily dependent on the quality of the dataset.\n\nIf every photo of a waterbird was in water, and every photo of a landbird was on land, did the model learn anything about discerning birds, or just about noticing the presence of water? If a duck was on a sidewalk, would the model declare it a landbird?\n\n**Mechanistic shaping** is Goodfire’s term for using a model’s internal representations to steer a model’s training objective towards more specificity. The **internal representation** (I often call it the *latent representation* though maybe a misnomer) is just the set of numbers that a model computed at some intermediate stage in its layers.\n\nInside the model’s internal representation, it might already have both the internal representation for both the shape of the bird and the water behind it. The standard loss function we would use (binary cross-entropy loss in this case) does not give meaningful signal during the training process that the model should focus on the shape of the bird rather than where it is.\n\n## Expression and Reliance\n\nGiven our bird classifier, we need to figure out two central questions: can we tell from the internal representation that there’s water in the photo? Goodfire would distinguish this as **expression**.\n\nSecondly, how much does the water influence the model’s inference? Goodfire calls this **reliance**. This is basically what I said above: find the representation, make the model learn to rely on it less/more during training.\n\nGoodfire used this waterbird classifier example in their paper<sup>[2](#fn:2)</sup>, and I wanted to try to recreate on a smaller simpler example the same phenomenon they are describing. I did a variation of the classic MNIST test, but with two digits next to each other.\n\nThe training objective is simple for the experiment: classify the left digit. The cheeky part of this is that both digits will be the same for our training set. So it could learn to just read the right digit to also minimize the loss.\n\nFrom the figure below you can see that the model was actually relying on the right digit, as if you change it the accuracy drops considerably despite the training labels being meant to represent the left digit.\n\n## How a measurement becomes a loss\n\nOne possible solution is to just train with a more balanced and representative dataset for the task–easy when its a simple experiment like this, but can be quite difficult on particular tasks! Instead, we can use reliance shaping.\n\nInstead let’s try this approach: we can first show the model two threes, then a three and an eight, and see which numbers change inside the model. If $h(3,8)$ is the internal representation for the left three paired with the right eight, the change from that swap is\n\n$$ \\Delta h = h(3,8) - h(3,3). $$\nWe repeat this process for all ten digits. Keeping the left image fixed, we subtract its average representation across the ten pairs from each pair’s representation. Then we average those shifts across left images for each right digit. Together those average shifts span what’s called a *concept subspace*, shown as the plane below.\n\nPick a direction in the plane and move one unit along it. Suppose the score for “three” rises by 2 points. If every score rises by 2, nothing changes about the prediction. But if the average score rises by only half a point, “three” gains $2 - 0.5 = 1.5$ points relative to the average. We’ll call that gain its *sensitivity* in this direction.\n\nThe classifier’s weights give us both score changes. For a step $v_j$, we use digit $c$’s weights $w_c$ to get its score change, and the average weights $\\bar w$ to get the average change.\n\n$$ S_{c,j} = \\underbrace{w_c \\cdot v_j}_{\\text{change for digit }c} - \\underbrace{\\bar w \\cdot v_j}_{\\text{average change}}. $$\nWe square the sensitivities so gains and losses can’t cancel, then average over the ten digits and nine directions. I added that penalty to the usual prediction loss with a weight of 0.1.\n\n$$ L = L_{\\text{prediction}} + 0.1 \\times \\operatorname{mean}_{c,j}(S_{c,j}^2). $$\nBack on our pair of threes, a confident “three” still earns a small prediction loss. But if the classifier’s weights let a change on the right tip the answer toward eight, it pays the extra penalty. The test is to give it a fresh pair where the digits disagree and see which one it reads.\n\n## Conclusion\n\nOn my experiment, ordinary training read the left digit correctly **46.8%** of the when the digits disagreed–showing that the model basically learned to read both digits about evenly.\n\nUsing reliance shaping, this reached **97.9%**, at a cost of about one percentage point on accuracy for matching pairs.\n\nOf course, I already had the labels and training data for the examples I had used to estimate the directions in the subspace. If we trained directly on that dataset by mixing it into the original training recipe, we get a classification accuracy of **97.3**.\n\nThis is pretty decent results, and though a little bit of a contrived and simple example, it does lend some creedence to the idea that you can bend training around certain representations for learning, with possible applications in safety, alignment, and who knows what else.\n\n## Citation\n\nIf you want to cite this post, you can use the following citation.\n\nMonty Bichouna. [An Intro to Mechanistic Shaping](https://nostalgia.dev/posts/intro-to-mechanistic-shaping/). November 16, 2026.\n\n```\n@misc{bichouna2026mechanisticshaping,\n  author = {Monty Bichouna},\n  title = {An Intro to Mechanistic Shaping},\n  year = {2026},\n  month = {November},\n  url = {https://nostalgia.dev/posts/intro-to-mechanistic-shaping/}\n}\n```\n\n1. Satchel Grant and Michael Byun. *[Shaping and Monitoring Models Through Their Internal Representations](https://www.goodfire.com/research/mechanistic-shaping).* Goodfire, November 5, 2026.[↩︎](#fnref:1)\n2. Thomas Fel et al. *[Shaping & Monitoring Models through their Internal Representations](https://arxiv.org/abs/2611.07071).* arXiv:2611.07071, November 4, 2026. Section 1 defines expression and reliance.[Section 2.1](https://arxiv.org/html/2611.07071v1#S2.SS1) describes the Waterbirds experiment.[↩︎](#fnref:2)[↩︎](#fnref1:2)", "url": "https://wpnews.pro/news/an-intro-to-mechanistic-shaping", "canonical_source": "https://nostalgia.dev/posts/intro-to-mechanistic-shaping/", "published_at": "2026-11-16 00:00:00+00:00", "updated_at": "2026-11-19 10:42:26.453541+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "ai-safety", "neural-networks"], "entities": ["Goodfire", "MNIST"], "also_reported_by": [], "takeaway": "Measure a concept's reliance by swapping inputs (e.g., a 3 next to a 3 versus a 3 next to an 8), averaging the resulting representation shifts across pairs to span a concept subspace, then shaping training along that direction instead of rebalancing the dataset.", "recommendation": {"advice": "When a model latches onto a spurious feature, compute average representation shifts from controlled input swaps to build a concept subspace, then add reliance shaping to the loss instead of only rebalancing the dataset.", "playbook": "training-data", "playbook_url": "https://wpnews.pro/playbooks/training-data"}, "alternates": {"html": "https://wpnews.pro/news/an-intro-to-mechanistic-shaping", "markdown": "https://wpnews.pro/news/an-intro-to-mechanistic-shaping.md", "text": "https://wpnews.pro/news/an-intro-to-mechanistic-shaping.txt", "jsonld": "https://wpnews.pro/news/an-intro-to-mechanistic-shaping.jsonld"}}