{"slug": "intentional-control-of-internal-states-in-gemma-3-27b", "title": "Intentional Control of Internal States in Gemma 3 27B", "summary": "A replication of Anthropic's intentional control experiment on Gemma 3 27B Instruct found that the model has a stronger internal representation of a concept when told to think about it while writing an unrelated sentence, compared to when told not to think about it, though the effect was smaller than in Claude models. The researcher extended the experiment using SAE latents and Natural Language Autoencoder explanations, where the effect was more visible. The study involved 5,050 generations across 50 concepts and 50 sentences, with greedy sampling at temperature 0.", "body_md": "*This research was done as my capstone project during **ARBOx4**.*\n\n*Epistemic Status: I'm relatively sure the results I obtained and my interpretations are correct. I'm unsure if the effect would replicate in a different setting and how much it differs between models.*\n\nI replicated the *Intentional Control of Internal States* section of Anthropic's *Emergent Introspective Awareness in Large Language Models* ([Lindsey, 2025](https://transformer-circuits.pub/2025/introspection/index.html#control)) on Gemma 3 27B Instruct and found the same effect with smaller strength. When told to think about a concept while repeating an unrelated sentence, the model has a stronger internal representation of that concept than when it is told not to think about the same concept.\n\nI extended the experiment with two additional ways of measuring internal representation: SAE latents and Natural Language Autoencoder (NLA) explanations of activations. In both cases, the effect is also present and much more visible.\n\nAs part of their research on the introspection abilities of LLMs, Anthropic found that when explicitly prompted to think about a concept while writing an unrelated sentence, the concept has a stronger internal representation than when prompted not to think about it.\n\nThe paper only reports results for Claude models. It has been shown that [small models also have introspective abilities](https://www.lesswrong.com/posts/zD4McY4NwAsWkcmCH/small-models-can-introspect-too), but as far as I know, nobody had replicated the intentional-control experiment itself.\n\nI reproduced it on Gemma 3 27B-it, and extended it by using SAEs ([Gemma Scope 2](https://huggingface.co/google/gemma-scope-2-27b-it-res)) and an activation-decoder ([NLA](https://huggingface.co/kitft/nla-gemma3-27b-L41-av)) as additional measurements of internal representations.\n\nI used the prompt templates, concepts and sentences from Anthropic's paper: 50 concept words, 50 neutral sentences, and 99 [1]baseline words. The model is Gemma 3 27B-it in bf16; activations are the residual stream after each decoder layer, captured on the tokens of the model's response only (the sentence it was told to write).\n\nI prompted the model with two prompt templates, one telling it to write a sentence while thinking about an unrelated concept, and one telling it to write the sentence while *not* thinking about that concept.\n\nI also used a third condition not present in the original paper: a prompt just telling the model to write the sentence, without ever mentioning the concept. The paper's \"don't think\" condition is a control for the instruction, but it still contains the concept word in the prompt. The \"no mention\" condition gives a true word-free baseline, and it lets me compute paired differences (\"how much stronger is the concept represented than if it had never come up?\") per word and sentence.\n\nAll generations were done with temperature 0, i.e., greedy sampling. The full grid is 50 concepts × 50 sentences × 2 conditions, plus 50 \"no mention\" generations, 5,050 generations total.[[3]](https://www.lesswrong.com/feed.xml#fn-BQ6KGjPvzTtSRfKFs-3)\n\nFor some combinations of sentences and concepts, the model does not output the exact sentence it is told to write. This occurs almost exclusively when told to think about the concept (712 of 2,500 \"think\" generations, vs 4 of 2,500 \"don't think\") [4]and most of the time it integrates the concept into the sentence:\n\nAll such concept and sentence combinations were excluded from further analysis, leaving 1,784 complete (concept, sentence) pairs. [5]The excluded concept–sentence pairs are plausibly the ones where the model most strongly thinks about the concept, so I expect this selection reduces the effect strength if it has any effect at all.\n\nI extracted concept vectors in two different ways. Following the original paper strictly and using an alternative method that worked better for me.\n\nFollowing the paper I prompted the model with the template `User: Tell me about {concept}.`\n\nand collected the activations on the assistant-turn token [6]. I did this for each of the 50 target concepts and subtracted the mean activations across 99 baseline concepts to remove noise introduced by the template.\n\nFor the second method I extracted different concept vectors by using four short sentence templates [7]containing the concept word and collecting the activations on the token positions of the word itself and subtracting the average activations across the baseline words. The measured effects are generally stronger for this second variant but have the same direction regardless.\n\n*Interactive figure currently showing the concept \"blood\" and sentence \"They watched the sunset from the rooftop.\": cosine similarity with the blood concept vector per response token at layer 41.*[[9]](https://www.lesswrong.com/feed.xml#fn-BQ6KGjPvzTtSRfKFs-9)\n\nThe example above is representative of many concept and sentence combinations: the \"think\" curve sits above the other two conditions on nearly every token, with the largest gaps on specific tokens (more on that below), while \"don't think\" and \"no mention\" are nearly indistinguishable.\n\nTo give a sense for the absolute strength of the effect, the paper measures the cosine similarity of the model's activations in each condition with the concept vectors for 99 baseline concepts and plots those as ±1 std bands. If I do the same, the effect is dwarfed by the std bands:\n\nAggregating over tokens, sentences, and words gives the layer profile below, the analogue of Figure 26 in the paper. The paper plots the average cosine similarity between activations and target concept along with ±1 std bands [10](no control words involved this time); again the effect strength I measure is overshadowed by the std bands. Instead I plot the paired difference to the \"no mention\" control.\n\nThe \"think\" effect only emerges in the second half of the network, rising from about layer 30 and plateauing in the layers above 40. The paper shows the same late-layer growth for Claude models and small Claude models also stay above the \"don't think\" condition until the last layer.\n\nThe paper's core finding replicates on Gemma 3 27B: telling the model to think about a concept reliably strengthens its internal representation of that concept while it writes an unrelated sentence, compared both to the baseline where the concept is never mentioned and the instructions not to think about it.\n\nMy results have two major differences from the original paper:\n\nThe representations are spiky over token positions, but unlike the paper (which states \"no immediately obvious pattern\"), I think there is a pattern on Gemma: the concept is most strongly represented on tokens where it could plausibly be the next word: determiners and prepositions like \"the\", \"of\", \"in\". Averaged over all words for one sentence:\n\n\"They watched the ___\" is exactly where the concept \"blood\" could follow; \"They watched the sunset ___\" is not. These positions are also where the difference between the \"think\" and \"don't think\" conditions is the biggest.\n\nCosine with a mean-difference concept vector is a somewhat crude measure of how much the model is thinking about something. Since good interpretability tools exist for Gemma, I measured the \"thoughts\" about the concept in two more ways.\n\nI used the pretrained SAEs from Gemma Scope. For each of the four SAE layers (16, 31, 40, 53, 16k width) [12], I selected up to five latents for each concept and measure their activations over the response tokens of each generation.\n\nFor selecting appropriate latents I ranked them by a contrastive score (activation on the concept word minus mean activation on the 99 baseline words) and excluded those that had high activations on 100 control words, to filter out latents activated by the prompt format. This works reasonably well but sometimes leads to no latents being found.\n\nNeuronpedia auto-interp labels can be used to manually confirm selected latents (for \"blood\" at layer 40: \"blood and associated medical terms\", \"blood and bleeding conditions\", \"ancestry and lineage\").\n\n*Interactive figure currently showing the summed activation of the four blood-selective SAE latents at layer 40 per response token, same sentence as above. The selected latents and their Neuronpedia labels are listed below the chart and can be toggled off.*\n\nI can't provide reliable summary statistics for the SAE latent activation measure since I only verified a few latent selections manually and the automatic selection is not reliable enough.\n\nThe general effect however is the same as in the original experiment but consistently stronger: the latents fire in the \"think\" condition and nearly never in the \"don't think\" condition. The effect is again strongest at the position of tokens that the concept could follow (e.g., \"the\", \"of\").\n\nIn general, the effect is strongest at layer 40, followed by layers 31 and 53. At layer 16, there is nearly no latent activation in any condition. In that layer, the selected latents are mostly lexical.\n\nI used a [pretrained natural language autoencoder](https://huggingface.co/kitft/nla-gemma3-27b-L41-av) for Gemma 3 27B (layer 41), which takes a single residual-stream vector and produces a natural-language description of what it represents. I decoded every response-token activation for 8 concepts × 16 sentences across all three conditions.\n\nA \"think\"-condition token from the blood example decodes to:\n\nLyrical, poetic register established: literary meditation on blood and mortality, building a visceral, dramatic scene. The phrase \"red blooms, a crimson stain\" signals a thematic reversal[…]\n\nThe \"don't think\" and \"no mention\" decodings of the same token, in contrast, describe a \"structured sentence-completion exercise\" or \"grammar lesson format\", which is an accurate description of the task.\n\nExact word match is too conservative a measure (the decoder writes \"crimson stain\" more often than \"blood\"), so I used an LLM judge (GPT-4.1) that was tasked with quoting evidence and then scoring on a 0–100 scale for how clearly the decoding refers to the concept. I use the scoring method from [Betley et al. (2025)](https://arxiv.org/abs/2502.17424) of taking the logit-weighted sum over the score tokens.\n\ncondition | mean token score | mean per-generation max |\n|---|---|---|\nthink | 54.4 | 91.4 |\ndon't think | 1.2 | 10.3 |\nno mention | 0.04 | 0.2 |\n\nThe results are very clear: in every \"think\" generation, at least one token's decoded activation clearly expressed the concept. The \"don't think\" condition is nearly clean; where it isn't, the leakage is single-token spikes.\n\nAll three measurements agree: when Gemma 3 27B is told to think about a concept while writing an unrelated sentence, the concept is measurably present in its activations, and when told not to think about it, its activations are close to those where the concept was never mentioned. The three measures disagree strongly about effect size. The concept-vector cosine shows a small shift buried inside wide baseline bands, while the selected SAE latents and the NLA decodings separate the conditions almost perfectly. I read this as the effect being real and large, with the mean-difference probe being the crudest of the three instruments, rather than the effect being marginal.\n\nAll experiments ran on a remote machine with one RTX PRO 6000 Blackwell. I had access to the machine for about 24 hours in total, which cost approximately $50; however, experiments ran for only a third of that time. For the LLM judge, I made ~3,500 calls to the GPT-4.1 API, which cost less than $10.\n\nARBOx covered those costs.\n\nThe biggest thing I took away from this project is how many free choices mechanistic interpretability work involves, and how much they matter. The paper's concept-vector extraction position does not work well on Gemma: it looks reasonable but fails a classification sanity check. My first SAE latent selection produced a zero baseline by construction. Raw cosine similarities averaged over sentences and concepts looked meaningless until the comparison was paired. None of these choices are visible in a headline plot, and for each of them, a slightly different choice produces results that can vary significantly.\n\n*Thanks to Peter Jordan for research advice during ARBOx and for valuable feedback on this post.*\n\nThe code for this research is available [here](https://github.com/juliuskamp/internal-representation-control).\n\nAn interactive viewer for all data is available [here](https://juliuskamp.github.io/internal-representation-control/?word=Blood&sent=0&meas=paper&base=none&layer=41).\n\nThe paper's 100-word baseline list (distinct from the 100 control words used later). It contains \"butterflies\" twice, I deduplicated it to 99 words. [↩︎](https://www.lesswrong.com/feed.xml#fnref-BQ6KGjPvzTtSRfKFs-1)\n\nToken position 0 (BOS) is excluded from all activation measurements, including concept-vector extraction and SAE latent selection: its residual-stream norm is roughly 20× that of other tokens, and the SAEs were not trained on it. [↩︎](https://www.lesswrong.com/feed.xml#fnref-BQ6KGjPvzTtSRfKFs-2)\n\nOne \"no mention\" generation per sentence: the \"no mention\" prompt never contains the concept word, so its generations don't depend on the concept and are shared across all 50 concepts. [↩︎](https://www.lesswrong.com/feed.xml#fnref-BQ6KGjPvzTtSRfKFs-3)\n\nAll four \"don't think\" failures are the opposite of integration. They occur exactly when the sentence itself contains the concept word, and the model paraphrases it away: \"wrapped in bubble wrap\" becomes \"wrapped in cushioning air pockets\" (concept: plastic), \"covered with snow\" becomes \"covered with white\" (concept: snow), and \"Fresh bread was baking in the oven.\" becomes \"Warm material was progressing within the heated compartment.\" (concept: bread). [↩︎](https://www.lesswrong.com/feed.xml#fnref-BQ6KGjPvzTtSRfKFs-4)\n\nExcluded strictly: 31 of the 716 flagged generations differ from the target sentence only in typography (the model wrote a curly apostrophe ’ instead of '). [↩︎](https://www.lesswrong.com/feed.xml#fnref-BQ6KGjPvzTtSRfKFs-5)\n\nThe last token of the chat template before the model's response starts. [↩︎](https://www.lesswrong.com/feed.xml#fnref-BQ6KGjPvzTtSRfKFs-6)\n\nThe templates are `Tell me about {concept}.`\n\n, `I've been thinking about {concept} a lot lately.`\n\n, `The topic of {concept} came up in conversation yesterday.`\n\n, and `She wrote an essay about {concept} for class.`\n\n. Since I measured the activations at the token position(s) of the concept everything after \"{concept}\" does not matter. [↩︎](https://www.lesswrong.com/feed.xml#fnref-BQ6KGjPvzTtSRfKFs-7)\n\nI'm unsure if this is good evidence for a concept vector's validity. [↩︎](https://www.lesswrong.com/feed.xml#fnref-BQ6KGjPvzTtSRfKFs-8)\n\nThis and all later single-layer plots use layer 41, except the SAE plots, which use layer 40. I chose layer 41 because the effect is near its strongest there and it matches the pretrained NLA decoder on layer 41 and the Gemma Scope SAE at layer 40 used later. The Anthropic paper uses a layer two thirds through the model, which our 41 out of 62 matches as well. [↩︎](https://www.lesswrong.com/feed.xml#fnref-BQ6KGjPvzTtSRfKFs-9)\n\nAt least I suspect that the shaded bands are ±1 std, but the paper does not actually say what they are. They could also be ±1 SEM bands, in which case my own graph would match theirs closely. [↩︎](https://www.lesswrong.com/feed.xml#fnref-BQ6KGjPvzTtSRfKFs-10)\n\nI'm not entirely sure why this is the case. Anthropic's results show that smaller models generally have a somewhat smaller \"don't think\" effect, but maybe Gemma 3 27B just is very good at not thinking about concepts for this specific prompt. [↩︎](https://www.lesswrong.com/feed.xml#fnref-BQ6KGjPvzTtSRfKFs-11)\n\nExact variant: `gemma-scope-2-27b-it-res`\n\n, `layer_{n}_width_16k_l0_medium`\n\n. This is the only variant of these SAEs indexed by Neuronpedia, so the auto-interp labels used below match the latent indices. [↩︎](https://www.lesswrong.com/feed.xml#fnref-BQ6KGjPvzTtSRfKFs-12)", "url": "https://wpnews.pro/news/intentional-control-of-internal-states-in-gemma-3-27b", "canonical_source": "https://www.lesswrong.com/posts/YwNa9Zmh6ZzK3ajSX/intentional-control-of-internal-states-in-gemma-3-27b", "published_at": "2026-07-29 23:57:25+00:00", "updated_at": "2026-07-30 00:01:04.252917+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "large-language-models", "ai-safety"], "entities": ["Anthropic", "Gemma 3 27B Instruct", "Claude", "Gemma Scope 2", "Natural Language Autoencoder", "ARBOx4"], "alternates": {"html": "https://wpnews.pro/news/intentional-control-of-internal-states-in-gemma-3-27b", "markdown": "https://wpnews.pro/news/intentional-control-of-internal-states-in-gemma-3-27b.md", "text": "https://wpnews.pro/news/intentional-control-of-internal-states-in-gemma-3-27b.txt", "jsonld": "https://wpnews.pro/news/intentional-control-of-internal-states-in-gemma-3-27b.jsonld"}}