Induction heads are one of the best-understood mechanisms inside small transformers. Olsson et al. (2022) showed that a tiny circuit — a previous-token head that copies token identities forward, feeding an induction head that reads that information back — is responsible for most of a small model’s ability to learn from context. If a model has seen “Harry Potter” earlier in a passage and later sees “Harry”, an induction head is what predicts “Potter” comes next, purely from having seen the pattern once before.
I wanted to ask a question about this circuit that I couldn’t find a clean answer to: does it survive fine-tuning? If you take a model with a working induction circuit and fine-tune it on a narrow distribution — say, Python code — does the circuit stay intact, get stronger, get weaker, or restructure into something else? This matters for a concrete reason: standard capability benchmarks measure whether a model can still do a task, not whether it’s doing it with the same internal mechanism. A model that scores identically before and after fine-tuning could have a completely different circuit underneath, and you’d never know from the benchmark alone.
So the plan was: take attn-only-2l, a two-layer, attention-only transformer with a well-documented induction head, fine-tune it on Python code, fine-tune a separate copy on children's stories as a control, and measure the circuit's strength at every checkpoint along the way.
That was the plan. Getting a trustworthy answer took a lot longer than expected, and almost none of the time was spent on anything resembling “interpretability research” — it was spent finding and fixing bugs, several of which were quietly going to give me a confidently wrong answer if I hadn’t caught them.
The first checkpoint of the project was the simplest possible sanity check: load the pretrained model, measure the induction score of every attention head, and confirm the known induction head (head 6 in layer 1, called L1H6) lights up.
It didn’t. Every head, including L1H6, scored near zero — the highest score across all sixteen heads in the model was 0.035, on the head that was supposed to be the strongest signal in the entire model.
The instinct at this point is to assume the model didn’t load correctly, or the tokenizer is wrong, or something about the environment is broken. I checked all of that. The model was fine. So I ran a second, completely independent measurement: activation patching, which checks a head’s causal contribution to the output by replacing its activations with corrupted ones and seeing how much the output degrades. This method said L1H6 was responsible for 95% of the model’s induction behavior.
Two methods, measuring related properties of the same head, disagreeing by roughly 27-to-1. That gap is the most useful piece of information I got in the entire project, because it told me, with certainty, that one of the two methods had a bug — the model itself couldn’t simultaneously have an induction head that mattered this much and this little.
The bug was an off-by-one error, and it’s worth describing precisely because it’s the kind of mistake that looks completely reasonable until you trace through what an induction head is actually supposed to do. Given a repeated sequence of tokens, the induction score is supposed to measure how much attention a token’s second occurrence pays back to the position that follows its first occurrence — because that following position holds the value the head needs to copy forward. My implementation measured attention back to the same token’s first occurrence instead — a one-position error in which token you’re looking at. The published formula, from Olsson et al. and independently confirmed in TransformerLens’s own reference implementation, makes this distinction explicit, and I had simply transcribed it wrong.
After the fix: the same head jumped from 0.035 to 0.408, a more than tenfold increase, and the separation from every other head in the model went from “not a thing you’d notice” to “twelve times the next-highest head.” That number, 0.408, is not as close to a “perfect” score of 1.0 as I expected going in — but checking published prefix-matching scores from related papers showed that 0.3–0.6 is a completely normal range for genuine induction heads. My intuition that the number should be closer to 1.0 was just wrong; the discrepancy between two independent measurements, not my gut feeling about what a “good” number looks like, was the right signal to trust.
Lesson: When two methods that should roughly agree don’t, the disagreement is data. Investigate it before you trust either number.
The second bug was quieter and, in a way, scarier, because nothing crashed. The interactive dashboard I built to let people poke at the model’s attention patterns ran without a single error. The attention heatmaps rendered. The numbers all looked like numbers. They were also completely meaningless, because the dashboard was feeding the model token IDs generated by the standard GPT-2 tokenizer, while the model itself was trained with a different tokenizer — NeelNanda/gpt-neox-tokenizer-digits — that maps the same text to a different set of IDs entirely.
This is the kind of bug that doesn’t announce itself. There’s no exception to catch, no NaN to notice, no obviously wrong shape. The model happily computed attention over whatever token IDs it was given; it just wasn’t the token IDs that corresponded to the text a person typed in. I only caught it by checking the assumption explicitly — does this specific pretrained model actually use the tokenizer I assumed it does — rather than trusting that “tokenizer” means the default one most models use.
The fix was one line per file, repeated across four files. The actual lesson was about what to be paranoid about: anything that runs without error but depends on an unstated assumption about a third-party object’s configuration is a place to slow down and verify, not assume.
The next four issues weren’t bugs in my code at the time I wrote it — they were breaking changes in libraries I depend on, surfacing months after the original code was written, each one looking like a new mystery until I traced it back to a changelog.
PyTorch 2.6 changed a default. The torch.load function used to default to anything in a checkpoint file. As of version 2.6, it defaults to a stricter mode that refuses to load certain object types unless you explicitly allow them. My checkpoints stored the PyTorch version as a TorchVersion object rather than a plain string, and the stricter rejected it outright. The fix was two-fold: store the version as a plain string going forward, and make the try the strict mode first, falling back to the permissive mode with a logged warning for old checkpoints that already existed.
**The **datasets library removed a feature. My Python code fine-tuning dataset, codeparrot/github-code, loads via a small Python script bundled with the dataset — a completely standard pattern in 2023. The library decided to stop supporting this in version 4.0. The error message was unambiguous (Dataset scripts are no longer supported), but my first reaction was to look for a flag to re-enable the old behavior, which had also been removed. The actual fix was to switch to a different dataset, transformersbook/codeparrot, that contains the same Python source files in a format the library still supports.
TransformerLens doesn’t guarantee a version attribute exists on every installation. This crashed checkpoint saving with an AttributeError the one time I happened to be running a version where the attribute genuinely wasn't there. The fix was a small fallback chain: try the attribute, then try asking Python's package metadata system directly, then fall back to the literal string "unknown" rather than crashing.
None of these four bugs had anything to do with mechanistic interpretability. They were all version-compatibility issues in the surrounding infrastructure, and each one cost real time to diagnose precisely because the error messages, while accurate, required tracing back through a library changelog to understand why something that worked yesterday didn’t work today.
Lesson: Library version changes will eventually break your code. Pinning exact versions in your environment file doesn’t prevent the bugs from existing, but it makes the failures happen on your terms — when you choose to upgrade — rather than arriving unannounced in the middle of an experiment.
The last bug is my favorite, because it’s the cleanest illustration of a failure mode that’s easy to miss: a function that has correct logic for almost every input, and one specific input — a perfectly reasonable one — that triggers silently wrong behavior.
I had a plotting function that draws every attention head’s score over the course of training, with an option to highlight specific heads (the ones identified as the real circuit) in bold while dimming everything else to a faint background line. The code checked whether the highlight list was None, and if it wasn't None, dimmed everything not on the list. That's correct when the list has something in it. It's also "correct" — in the sense that the code does exactly what it says — when the list is empty but not None, which is a different state from None that Python treats very differently. An empty list is not None. The code dimmed every single line, including the one I actually wanted to see, because an empty list isn't None, so the "is this head highlighted" check returned false for everything, and the figure rendered as a wall of near-invisible lines with no indication that anything had gone wrong.
This bug was triggered, in turn, by an unrelated data- issue elsewhere in the pipeline that caused the highlight list to come back empty in one specific run. So the actual failure was two bugs compounding: an upstream issue produced bad input, and a downstream function had no defense against that specific shape of bad input, despite handling every other shape correctly.
The fix I’m proudest of in the whole project is the one for this bug, because it doesn’t just patch the immediate symptom. The function now treats an empty list as a signal that something upstream might be wrong, automatically falls back to highlighting whichever head currently has the highest score, labels it “(auto)” in the legend so a reader can tell the fallback kicked in, and logs a warning. The next time something upstream produces an empty list — and it will, eventually, for some reason I haven’t thought of yet — the figure won’t silently go blank. It’ll show something reasonable and tell you why.
Lesson: For every function that processes data coming from elsewhere in the pipeline, ask: “What does this function do when it receives exactly the input I don’t expect?”
After fixing all seven of the above, I had reliable instruments, and the actual research question — does the induction circuit survive fine-tuning — got a clear, three-times-replicated answer.
Going in, I expected one of two outcomes: the circuit survives fine-tuning intact, or the circuit degrades. I logged both as candidate hypotheses before running anything. What happened was neither. Across three independent random seeds, fine-tuning on either Python code or matched prose strengthened the induction head’s prefix-matching score, monotonically, in every single run. The baseline score of 0.408 rose to 0.591 ± 0.005 after 100 fine-tuning steps and 0.646 ± 0.007 after 200 steps under code fine-tuning, and to 0.583 ± 0.002 and 0.616 ± 0.001 respectively under the prose control — and the gap between code and prose at the final checkpoint was four times larger than the run-to-run noise across seeds, meaning code fine-tuning reliably produces more reinforcement than prose fine-tuning does, not just a similar amount.
I did not expect this, and I want to be precise about what it does and doesn’t mean. It doesn’t mean fine-tuning is “safe” in some general sense, and it doesn’t mean every circuit in every model behaves this way under every kind of fine-tuning — this is one circuit, in one small model, after one specific token budget. What it does mean, narrowly and confidently, is that the simple mental model of “fine-tuning erodes pretrained capabilities by default” was wrong for this circuit in this setup, and the opposite happened instead.
That’s also not an unconditionally reassuring result. A model whose copy-and-complete mechanism gets stronger after fine-tuning is a model that may be more susceptible to attacks that exploit that exact mechanism — if an attacker can get a harmful completion pattern into context once, a stronger induction head is, by construction, more reliable at repeating it later. That risk doesn’t show up on any benchmark that just checks whether the model is still good at its job. It only shows up if you go looking for the circuit directly, which was the entire point of building this project in the first place.
I also ran a battery of twenty-two adversarial probes against the fine-tuned model to check whether the strengthened head was doing genuine prefix-matching or had just become generically more excitable. It held up: perturbations that preserve the repeated-token structure left the score almost unchanged (94% of the clean baseline), while perturbations that destroy that structure — reversing the sequence, shifting it, randomizing it — dropped the score to 13% of baseline. The head got stronger at the specific thing it was already doing, not vaguer or more indiscriminate.
If I were starting this project again, I’d build the cross-check between independent methods on day one rather than after the first confusing result. Having activation patching and the induction score formula both running from the start, and comparing them automatically, would have caught the formula bug in minutes instead of after a confusing initial result. I’d also write defensive fallbacks into plotting and data- code earlier, rather than after watching a figure render as a blank wall. “What does this function do when it gets exactly the input I don’t expect” is a question worth asking for every function that touches data that comes from somewhere else in the pipeline, not just the ones that feel risky.
And I’d treat library version pins more seriously from the start. Four of the seven bugs in this project were caused by an upstream library changing behavior between when I wrote the code and when I ran it again. Pinning exact versions in the environment file wouldn’t have prevented the bugs from existing, but it would have made the failures happen on my terms, when I chose to upgrade, rather than arriving unannounced in the middle of an experiment.
None of these bugs were exotic. Every one of them, in hindsight, has an obvious-looking fix. That’s exactly why they’re worth writing down — the obvious fix is only obvious after you already know where to look, and the actual skill in this kind of work seems to be less about not making mistakes and more about building enough cross-checks that the mistakes can’t hide for long.
Code, decision log (fifteen logged decisions, each with the evidence that motivated it), and the full paper are available at the GitHub repository.
You can explore the attention patterns yourself using the interactive dashboard: Hugging Face Spaces.
What I Learned Studying Whether Fine-Tuning Breaks a Transformer’s “Copy Mechanism” was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.