An attention map can look like a copying algorithm. The bright diagonal is an observation: at each repeated token, one head attends to the token that followed its earlier match. That image alone cannot tell us what information the head writes, whether the write favors the copied token, or whether the model needs the head to perform the behavior.
This drawing climbs those questions in order. Pattern evidence locates a candidate. Logit attribution asks whether its output points toward the expected next token. Intervention removes or replaces the component and measures whether the behavior changes. Each layer supports a stronger claim while adding assumptions of its own.
The worked example is a real CPU export from the 124-million-parameter GPT-2 checkpoint, revision 607a30d…
, over three repeated-token prompts. The export script and page data record the prompt set, one tokenized display prompt, three display heads’ attention tensors, attribution, eight ablation conditions, versions, and selection rule. It is a small replication of an induction-like circuit, bounded to this model, selector, prompt set, and metric.
A head routes one write into a shared stream #
An attention head has two coupled jobs. Its QK circuit decides where to read. Its OV circuit decides what to write. The compact account in The Transformer §03 derives scaled dot-product attention [1];
§04shows how many heads run in parallel. Here the same machinery becomes an object of investigation.
For residual-stream matrix **X**, one head projects queries **Q** = **XW** Q, keys **K** = **XW** K, and values **V** = **XW** V. The causal mask removes future keys. Softmax turns the remaining QK scores into a row of weights. That row mixes values, and **W** O maps the result back to model width.
head(**X**) = softmax((** XW**Q)(** XW**K)T / √dh + **M**) **XW** V **W** O
The attention matrix belongs to the routing half only. Two heads can attend to the same source position and write different directions because their OV matrices differ. A head can also write something useful through a diffuse pattern, or aim a sharp pattern at information that later computation cancels.
Usually no. “Head” names an architectural component, not a guarantee of one semantic role. A head can participate in several behaviors, and one behavior can be distributed across several heads and MLPs. Interpretability supplies testable descriptions at a chosen level; it does not turn every parameter group into a tidy concept.
Interpretability turns weights into mechanism claims #
A trained model gives us weights, activations, and outputs. A mechanism claim adds structure: component A detects a condition, passes information to component B, and changes behavior C. The claim should predict what appears inside the model and what happens when the proposed route is disrupted.
The transformer-circuits framework [2] treats the residual stream as a shared communication channel. Heads and MLPs read from it and write back to it. This view makes composition concrete. A late head can read a feature written by an earlier head; its QK circuit and OV circuit can be studied as products of weight matrices rather than as isolated parameter names.
Three evidence layers keep the language disciplined. First, observe an activation or attention pattern associated with the behavior. Second, read the component’s output into a behavior-relevant basis, such as vocabulary logits. Third, intervene at the proposed component or path and compare the measured behavior. Correlation enters at the first layer. Causal evidence begins with a controlled change at the third.
. Component count and component necessity are separate facts.
[3]The ladder also blocks a common category error. A visualization [4] answers “where was weight placed?” A causal statement answers “what would have changed under an intervention?” The same experiment can produce a beautiful pattern and a weak, reversed, or prompt-dependent ablation effect. Those disagreements are results, not blemishes to hide.
A mechanism hypothesis becomes useful when it risks failure. “This head pays attention to repeated material” can absorb almost any heatmap after the fact. “[A][B] … [A] routes to the earlier [B], writes toward B, and loses that copying effect when the route is removed” predicts three measurements before the tensors are opened. It also predicts controls: break the repeated prefix and the stripe should weaken; preserve the prefix while changing the continuation and the copied target should change; patch the clean route into a corrupted run and the score should recover.
Candidate discovery and hypothesis testing therefore deserve separate data whenever the budget allows. One prompt set can locate promising heads, while a held-out set estimates whether the declared pattern and effect generalize. This page uses the same three tiny prompts for screening and reporting, so its claim stays at replication scale. The selector is published to make that limitation inspectable.
Induction continues a pattern seen inside the prompt #
Take a sequence A B C D … A B
. After the second B
, a model that uses the earlier pattern can raise C
as the next token. More generally, when a token A appears again, an induction pattern attends to the token B that followed the earlier A, then writes in a direction that helps predict B now. The behavior has the form [A][B] … [A] → [B]
.
This is a form of in-context pattern completion. The relevant association lives in the current sequence; no weight update occurs during the prompt. Olsson and collaborators [5] studied heads with this pattern across small attention-only models and larger language models, connecting their emergence to a sharp change in in-context learning during training.
Induction heads became a useful case study because they connect a visible algorithm, a compact weight-level account, and a behavioral capability. They also expose the limits of that connection. The training evidence in the original study associates induction-head formation with improved in-context loss; it does not reduce every instance of in-context learning to this one circuit. The repeated-token behavior gives us a tractable microscope slide, not a catalog of everything a language model learns from context.
induction target at position t = token at (previous occurrence of xt) + 1
The exact tokenization matters. Our prepared prompts use GPT-2 tokens that repeat as two equal eight-token cycles: letters, number words, and color words. The behavior metric averages, across the second cycle, the expected continuation token’s logit minus the mean logit of the other cycle tokens. That local comparison isolates copying within the prepared alphabet; it is not a language-model benchmark.
It is a controlled fragment of in-context learning: the prompt supplies a relation and the model continues it. Few-shot tasks can require richer representations, task inference, and several circuits. The repeated-token test is valuable because its expected route and next-token effect can be specified exactly.
Two heads compose the induction route #
The canonical two-layer account begins with a previous-token head. At position i, it attends to i − 1 and copies information about the previous token into the current residual state. That write gives a later head a way to compare “the token before me” with tokens at earlier positions.
The later induction head uses that written feature in its QK circuit. Its query at the current position matches keys at positions whose previous token resembles the current prefix. The matching key position is the token after the earlier occurrence. Its OV circuit then copies information about that source token toward the vocabulary direction for the next prediction.
Written as a path, the first head supplies a shifted token feature; the second head uses a QK composition to route toward the earlier continuation and an OV composition to write that continuation. The circuit is distributed through the residual stream. There is no wire connecting two named modules, only compatible reads and writes learned in separate layers.
Composition is a hypothesis about information flow, not a guarantee that these two heads work alone. Other heads establish token identity, MLPs transform the stream, and later layers can amplify or suppress the write. The useful experimental question is whether the named path predicts effects that survive controls.
Layer one: locate the candidate pattern #
Pattern evidence begins with a declared template. For every query in the second cycle, score attention to the position immediately after the corresponding token in the first cycle. Average that weight across eligible positions and prompts. This produces a prefix-matching score without asking whether the head’s output helps the expected token.
Our exporter computes that score for all 144 GPT-2 heads on three prompts. It screens the 24 highest heads, requires mean pattern score of at least 0.10, and retains candidates whose later ablation reduces the separately defined copying metric. Among those retained candidates, L5H1 had the largest aggregate drop. L5H5 actually had the brightest mean pattern, 0.599 versus 0.536, which illustrates why selection and evidence layers should remain visible.
The protocol pins the model repository and resolved commit, loads the checkpoint in evaluation mode with eager attention, and seeds Torch with zero. Each visible prompt is checked after tokenization: its first eight token IDs must exactly equal its second eight. Attention scores are averaged over seven valid continuation positions per prompt. That ledger prevents a whitespace split or multi-token word from quietly changing the intended circuit.
The selector mixes discovery and intervention because this page needs one candidate for the interactive. It records the full rule rather than presenting L5H1 as a famous coordinate chosen in advance. A stricter study would preregister the threshold, separate discovery prompts from evaluation prompts, repeat across seeds or checkpoints, and publish confidence intervals over a broader distribution of sequences.
At the demo’s active query, token G
in the second cycle, L5H1 places 0.783 attention on H
in the first cycle. That is the prescribed source for predicting the next H
. The row looks exactly like copying. The proper layer-one claim remains narrow: this head routes strongly along the induction template on this input.
0␠B
1␠C
2␠D
3␠E
4␠F
5␠G
6␠H
7␠A
8␠B
9␠C
10␠D
11␠E
12␠F
13␠G
14␠H
15
Use the head buttons to compare attention rows. Then zero the previous-token head, candidate induction head, or same-layer MLP. The bars never change with those toggles because they show the intact exported attention pattern for the selected display head. The behavior readouts do change: each toggle selects one of eight precomputed full forward passes. The cell reports a routing weight, not the value vector carried along that route or the effect of the resulting residual write. Attention also branches across heads and layers. A bright cell can carry irrelevant content, be canceled later, or accompany a behavior produced through another path.
Layer two: read the head in vocabulary space #
Logit attribution asks whether the head’s write points toward the token the copying hypothesis predicts. Take the selected head’s pre-projection result, pass only its 64-wide slice through the corresponding rows of W O, and obtain a 768-wide residual write. Then apply a declared readout into vocabulary space.
Our export uses the intact run’s final layer-normalization scale and gain, excludes bias, and dots the head write with GPT-2’s tied unembedding. This linear decomposition assigns L5H1 a direct contribution of +0.243 to the H
target logit at the active position. Relative to the mean contribution for the other cycle tokens, its direct copying-score attribution is +0.398.
DLAh→v = W U,v · LNscale,intact(z h W O,h) Layer normalization makes attribution conventions matter. Freezing the intact scale yields an additive accounting view; recomputing normalization for a component alone answers a different question. Nonlinear downstream computation also means a direct attribution cannot equal the result of removing the head and rerunning the network. Attribution supports “the write points toward H under this readout.” It does not support “the model requires this head.”
Raw logits and probabilities answer different practical questions. A logit difference remains readable when softmax saturates, while probability includes competition from the entire 50,257-token vocabulary. At the active position, both intact and ablated runs still favor H
strongly, yet the local copy score falls by more than three. Reporting the target probability beside the logit contrast shows why an apparently small probability change can accompany a material circuit effect.
Attribution can be expanded into a table over vocabulary tokens. A copying OV circuit should tend to place weight on the attended token’s own output direction across many source tokens. That weight-level “copying score” is related to, but distinct from, the prompt-level logit contrast used here. The page keeps the terms explicit so a matrix statistic, a component attribution, and a behavioral score do not collapse into one number.
Layer three: break the proposed route #
An intervention changes an internal variable and reruns the model. The simplest version here is zero ablation: before the selected attention output projection, set that head’s 64-value slice to zero at every position. The remaining eleven head slices pass through unchanged. A separate hook can zero the earlier previous-token head or the layer-five MLP output.
With all components intact, the active-position copying score is 9.418. Zeroing L5H1 reduces it to 6.218, a drop of 3.200. Across every eligible position in all three prompts, the mean drop is 1.023. Zeroing the earlier L4H11 head at the displayed position reduces the score by 2.350. These results causally implicate both components in this measured copying behavior under the stated zero intervention.
H
; L5H1 ablation assigns 92.54%. The logit contrast is a cleaner local measure here because the probability is already near saturation.The MLP control exposes interaction. Zeroing layer five’s MLP raises the active copying score by 1.405, while zeroing both that MLP and L5H1 leaves the score 0.100 above intact. Effects do not add independently. Removing a large component can shift layer normalization and downstream routes, so the ablation result belongs to the whole counterfactual run.
Zero is a strong and possibly off-distribution replacement. Mean ablation substitutes an average activation. Resampling ablation borrows a value from a control distribution. Activation patching replaces a corrupted-run activation with its clean-run counterpart and asks how much a clean behavior is restored. Path patching narrows the replacement to a proposed sender–receiver route. Each intervention defines a different counterfactual.
A clean–corrupted pair makes restoration especially legible. Start with a repeated sequence where the model predicts the expected continuation. Corrupt one prefix token while holding length and token class stable, then verify that the copying metric falls. Patch the candidate activation from the clean run into the matching site of the corrupted run. Recovery supports the claim that information carried at that site is sufficient to restore part of the behavior in that paired setting. Patching every site until something recovers can overfit as easily as scanning every heatmap, so the search procedure still belongs in the evidence record.
Necessity, sufficiency, and mediation are different claims. Ablation probes necessity under a replacement. Restoration probes a form of sufficiency. A path-specific experiment asks whether the sender’s effect reaches behavior through a named receiver rather than through every downstream consumer. None of these tests identifies a component’s natural-language “purpose”; each measures a counterfactual defined by the investigator.
Good controls include heads with similar attention entropy, nearby heads without the target pattern, shuffled continuation tokens, prompts with no repeat, and several token families. Report adverse cases, including candidates whose ablation changes the metric in the opposite direction. A pattern-only reading would assign those cases the wrong causal direction.
No. It shows that the measured behavior depends on the altered component under this replacement, to the observed degree. Redundant routes can preserve some behavior, and zeroing can disturb computations beyond the hypothesized path. Sufficiency needs a separate test, such as restoring or transplanting the proposed activation into a suitable control run.
- ▸QK and OV circuit plate with the induction composition
- ▸Claim-strength ladder: pattern → attribution → intervention
- ▸Intervention checklist and tensor-shape ledger
- ▸Blank experiment record for your own model runs
The result is local; the open problem is distributed #
This replication supports a bounded statement: GPT-2 contains heads that match a prepared induction template; one selected head writes positively toward the copied token under a frozen-scale logit readout; and zeroing that head reduces the declared copying score across the prompt set. It does not establish that every in-context behavior uses induction heads, that L5H1 has one exclusive function, or that the two named heads form a complete circuit.
Superposition is a central boundary. A model can represent more features than it has dimensions by placing features in overlapping directions. Polysemantic components and distributed features make one-head labels lossy. Anthropic’s toy models [6] show how sparsity and feature geometry can produce superposition; they provide a conceptual model, not a direct decomposition of GPT-2’s residual stream.
The same boundary applies to vocabulary attribution. A positive token direction is legible because the unembedding supplies a convenient basis. Many useful intermediate features have no single-token name. Sparse autoencoders and related feature-discovery methods try to recover more interpretable directions, but they introduce training objectives, dictionary size, reconstruction error, and feature-splitting choices. They remain outside this drawing’s circuit claim.
Causal tracing work such as ROME [7] locates states whose restoration can recover a factual prediction after a corrupted input. Circuit studies such as the GPT-2 indirect-object-identification work
combine path patching, component analysis, and adversarial tests over a richer behavior. Those methods extend the same principle: write a hypothesis that predicts an internal route, intervene on that route, and measure a task-relevant effect.
[8]607a30d…
, Transformers 4.57.6, Torch 2.8.0, Python 3.9.6, the prompt set, selection rule, three display heads’ tensor rows for one prompt, eight intervention conditions, and the script SHA-256.Several open problems remain. Candidate selection can overfit a prompt set. Interventions can move activations off distribution. Effects can be nonlinear and redundant. A circuit found in one checkpoint can drift during training or change across scales. Feature-level accounts can conflict with component-level accounts. The strongest work publishes the selector, inputs, metric, tensors, controls, and failures so another investigator can attack the claim rather than admire the diagram.
A useful next experiment would split dozens of token cycles into discovery and evaluation sets, include nonrepeating and repeated-prefix/wrong-continuation controls, and compare zero, mean, and resampling ablations. It would patch L4H11’s output specifically into L5H1’s key input, test nearby heads as negative controls, and bootstrap the copying-score effect across prompts. Repeating the protocol across several GPT-2 revisions or independently trained small models would separate one checkpoint’s coordinates from a recurring circuit motif.
Another extension would trace training checkpoints. If the proposed composition causes an in-context learning change, the previous-token signal, induction stripe, vocabulary attribution, and causal effect should emerge in a coherent order. A matched learning curve can test that temporal prediction. A final behavioral score alone cannot reveal whether the same internal route appeared suddenly, accumulated gradually, or was replaced later in training.
That standard is the durable lesson. Attention maps are useful maps. Attribution connects a component to an output direction. Intervention supplies the counterfactual evidence required for a causal mechanism claim. Keeping those layers separate makes interpretability results easier to reproduce, challenge, and improve.
- [01]
[Vaswani et al., ‘Attention Is All You Need,’ NeurIPS 2017](https://arxiv.org/abs/1706.03762)[↩](#cite-vaswani-2017) - [02]
[Elhage et al., ‘A Mathematical Framework for Transformer Circuits,’ 2021](https://transformer-circuits.pub/2021/framework/index.html)[↩](#cite-elhage-2021-framework) - [03]
[Michel, Levy, and Neubig, ‘Are Sixteen Heads Really Better than One?,’ NeurIPS 2019](https://arxiv.org/abs/1905.10650)[↩](#cite-michel-2019) - [04]
[Vig, ‘A Multiscale Visualization of Attention in the Transformer Model,’ ACL 2019](https://arxiv.org/abs/1906.05714)[↩](#cite-vig-2019) - [05]
[Olsson et al., ‘In-context Learning and Induction Heads,’ 2022](https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html)[↩](#cite-olsson-2022) - [06]
[Elhage et al., ‘Toy Models of Superposition,’ 2022](https://transformer-circuits.pub/2022/toy_model/index.html)[↩](#cite-elhage-2022-superposition) - [07]
[Meng et al., ‘Locating and Editing Factual Associations in GPT,’ NeurIPS 2022](https://arxiv.org/abs/2202.05262)[↩](#cite-meng-2022) - [08]
[Wang et al., ‘Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small,’ ICLR 2023](https://arxiv.org/abs/2211.00593)[↩](#cite-wang-2023-ioi)
Anything on this sheet still unclear — or anything you were too polite to ask out loud? File an RFI. Answers come from the drawing itself and cite their sheet numbers, and every question is recorded in the drawing log so the next revision can answer it in print.