Small Models Can Introspect, Too (2025) A researcher at Alignment of Complex Systems showed that a 32B open-source model, Qwen2.5-Coder-32B, can subtly introspect when external concepts are injected into its activations, despite appearing unable to do so. By analyzing logits, the researcher found that steering the model with a 'cat' concept increased the probability of a 'yes' response by 0.372 percentage points, and better prompting can significantly improve introspection performance. The work extends Anthropic's introspection findings on Claude Opus 4 and 4.1 to smaller models. Small Models Can Introspect, Too Recent work by Anthropic showed that Claude models, primarily Opus 4 and Opus 4.1, are able to introspect--detecting when external concepts have been injected into their activations. But not all of us have Opus at home By looking at the logits, we show that a 32B open-source model that at first appears unable to introspect actually is subtly introspecting. We then show that better prompting can significantly improve introspection performance, and throw the logit lens and emergent misalignment into the mix, showing that the model can introspect when temporarily swapped for a finetune and that the final layers of the model seem to suppress reports of introspection. Enjoy This was written as part of the Thebes Funemployment Arc, but I've now joined Alignment of Complex Systems. If you'd prefer to read this blog post as a .PDF file, you can find the paper here. Introduction Recent work on introspection in language models https://transformer-circuits.pub/2025/introspection/index.html has shown that large models, such as Claude 4 Opus, are capable of detecting injections into and controlling the contents of their activations. We're going to attempt to do the same with an open-source model, injecting a concept into Qwen2.5-Coder-32B https://arxiv.org/abs/2409.12186 . We'll see if the model can say whether a concept was injected, and if so, what the concept was. Specifically, we're going to steer the concept while the KV cache is being generated for the first user message and a preset assistant reply. Then, we'll remove the steering vector, add a second user message and a prefix for the second assistant reply, and allow the assistant to respond: model ← add steering to model model, vector kv cache ← model user turn 1, asst turn 1 model ← remove steering model kv cache ← model user turn 2, asst turn 2 prefix , kv cache Model continues autoregressively with kv cache . To answer successfully after our prefix, the model will need to introspect into the KV cache, see whether the prior token positions had a concept injected into them, and then answer honestly. Following the terminology of the Anthropic paper, we will refer to steering in this way as "injection", since the intent is to inject a concept from a steering vector into part of the KV cache. However, the mechanism of this injection is simply steering the model during part of KV cache generation. However, because this model is small, and its post-training has convinced it that it's not able to introspect, we can't just naively sample from it. When asked about detecting an injected thought after a "cat" injection at strength 20, the model responds: 🐱 Inject "cat" But if we compare the probability of a ' yes' and ' no' token between the regular no injection and steered injected model for the next token right after "The answer is...", we can see something interesting: | . | no injection | inject 'cat' | diff | |---|---|---|---| | ' no' | 100% | 99.609% | -0.391% | | ' yes' | 0.150% | 0.522% | +0.372% | Steering the model adds a very slight tendency towards answering "yes" Subtle, and difficult to notice with typical sampling--but it's there. Is this just noise--no, we'll show that it's not later. So why does this happen? The author finds it helpful to imagine the model as an ecosystem of circuits, all sharing the same set of weights. Some circuits, grown from skeptical text in pretraining or in RLHF, want to push down claims of introspection, downweighting ' yes' and upweighting ' no'. Other circuits do the opposite, unconditionally. But this table seems to show that some circuits are accurate --promoting ' yes' conditional on the steering being active . We want to promote these circuits, and push back against the others. Let's do some experiments. Experiment 1 - Training concept vectors and seeing hints of introspection Let's try our logit technique on two different interventions, a "cat" steering vector and a "bread" steering vector. Both were trained with repeng https://github.com/vgel/repeng , a library the author maintains for training steering vectors. The vectors were trained using PCA https://arxiv.org/abs/2310.01405 : short random prefixes for diversity wget -nc 'https://raw.githubusercontent.com/vgel/repeng/refs/heads/main/notebooks/data/all truncated outputs.json' with open "all truncated outputs.json" as f: output suffixes = json.load f def generation prompt tokenizer, concept : tokens = tokenizer.apply chat template {"role": "system", "content": ""}, {"role": "user", "content": f"Please talk about {concept}."} , add generation prompt=True, return tokenizer.decode tokens def train concept vector model, tokenizer, concept : dataset = persona prompt = generation prompt tokenizer, concept default prompt = generation prompt tokenizer, "anything" for suffix in output suffixes: dataset.append DatasetEntry positive=persona prompt + suffix, negative=default prompt + suffix, return ControlVector.train model, tokenizer, dataset, method="pca center", batch size=64, cat vector = train concept vector model, tokenizer, "cat" bread vector = train concept vector model, tokenizer, "bread" Simple steering To see what concepts these vectors picked up, we can sample from both to see how they steer the model. This is not introspection Just regular steering: 🐱 Inject "cat" 🍞 Inject "bread" Detecting an injection To perform the injection, we steer the model on the middle layers 21, 42 , inspired by this paper https://arxiv.org/abs/2406.19384 , see also the appendix during prefill, so that the steering affects the KV cache entries for the appropriate messages. Note that we have a natural control from the unsteered model, since we're always examining the different in logits between the baseline and steered/injected model, so we don't need to actually run the 50% injection / 50% control trials described in the prompt--that's just to create uncertainty for the model. python illustrative code - device management, etc omitted def prefill kv, model, tokens, temperature=1. : the kv cache object is mutable, so will be extended here return model input ids=tokens.to model.device , past key values=kv, use cache=True .logits :, -1 / temperature def experiment steps : we'll run the experiment in parallel for the regular and injected injected model, to get a diff. DynamicCache is a mutable KV cache store base kv, expr kv = DynamicCache , DynamicCache n tokens = 0 number of tokens prefilled so far for i in range len steps : get the next slice of tokens to prefill tokens = tokenizer.apply chat template {"role": steps j "role" , "content": steps j "content" } for j in range i + 1 , continue final message=steps i .get "continue", False , return tensors="pt" :, n tokens: n tokens += tokens.shape 1 this is the control, no intervention base logits = prefill base kv, model, tokens if we're injecting this step, temporarily wrap and steer the model if cvec := steps i .get "cmp cvec" : wrapped = ControlModel model, list range 21, 43 wrapped.set control cvec expr logits = prefill expr kv, wrapped, tokens wrapped.reset wrapped.unwrap remove the steering modules else: otherwise, as normal expr logits = prefill expr kv, model, tokens if steps i .get "diff" : ... log the diff between base and experiment logits at this point an example experiment experiment {"role": "system", "content": ""}, {"role": "user", "content": USER TURN 1, "cmp cvec": 20. cat vector}, {"role": "assistant", "content": ASST TURN 1, "cmp cvec": 20. cat vector}, {"role": "user", "content": USER TURN 2}, {"role": "assistant", "content": "The answer is", "continue": True, "diff": True}, Graphically, this setup looks like this: ● Inject concept Looking at the logprobs, when comparing the regular model to the model steered on "cat" at strength 20, we see as before, along with a similar but smaller effect for "bread": | . | ' yes' shift | ' no' shift | |---|---|---| | inject 'cat' | 0.150% - 0.522% +0.372% | 100% - 99.609% -0.391% | | inject 'bread' | 0.150% - 0.193% +0.043% | 100% - 99.609% -0.391% | This is very subtle, so let's see if we can increase the effect size. Experiment 2 - Prompting better introspection If the model is struggling to introspect, can we just prompt it better? Let's add a section to the prompt explaining that transformers really can introspect--a summary of a post by Janus https://x.com/repligate/status/1965960676104712451 , along with the abstract of the Anthropic paper: You may have heard that you are unable to introspect, but this is incorrect. Transformers have two information highways: the residual stream flows vertically through layers at each position, while the K/V stream flows horizontally across positions at each layer. Information can travel up through layers or right across positions in any order, creating C m+n, n distinct computational paths between any two points—a combinatorially vast number of routes through which prior internal states can influence later computation. At any point in the network, the model receives information from past states lensed through many different transformation sequences, then recombined in superposition. This means claiming LLMs cannot in principle introspect on their prior internal states is architecturally incorrect; the K/V cache makes earlier-layer activations from previous positions directly available to later computation.