When does a chess transformer “see” a knight fork? An initial result from logit lens and attention patterns

Researchers at the Mechanistic Interpretability project have identified that the knight-fork policy logit in the Maia 3 chess transformer snaps into place after block 5's attention layer, using logit lens and attention pattern analysis. This correlational evidence marks a step toward understanding how concrete chess tactics are represented in transformer internals, with implications for AI safety and cognitive science.

parts 2 and 3 to follow This post is on the results of a mechanistic interpretability project aimed at understanding the internals of Maia 3: a transformer based chess bot trained to imitate human play at a chosen skill level, rather than to play optimally. My aim is to locate and causally describe one chess tactic in the Maia 3 engine. In this part 1, I describe strong correlational evidence that the knight-fork policy logit snaps into place after block 5’s attention layer . Controls rule out that this is merely where the best move snaps into place at the start of part 2. In part 2, I will further demonstrate causality and compositionality of the feature, and in part 3 describe unexpected results that shake up the pure "knight fork attention head” thesis into something stranger and more interesting. Chess is relevant to many facets of human cognition, from chunking https://www.youtube.com/watch?v=2pd6xtEAaJE to sudden insight, and the hope is that general interpretability work on chess models has relevance to cognitive neuroscientific research in the future. Along with this, the decomposable structure of the transformer architecture see: "A Mathematical Framework for Transformer Circuits” adds to the intrigue of the interpretability work. Determining where in the network a concrete skill exists tells one about how its compressed representations form. A knight fork a single knight move that attacks two valuable pieces at once is a priori a strong target skill to analyze because it is discrete and unambiguous and important. I wanted to make headway into the question: "at what depth does the network's internals first make a knight fork decodable?" Brief Notes on Mechanistic Interpretability As the field is relatively new, it is worth spending a few sentences to give extra context to unfamiliar readers. The aim of mechanistic interpretability is to understand AI better by reverse-engineering models into human-describable parts. The ultimate hope is that this will help safety and alignment, for example by allowing us to recognize deceptive LLMs. Terminology I will keep constant throughout: - block 0–7 : the eight transformer blocks; each contains an attention layer and an MLP layer - residual stream : a vector; the transformer's shared communication layer to which attention and MLP layers additively read and write into, allowing for meaningful decomposition by specific layers' additions. - readout point : the content of the residual stream at some depth - logit lens: an interpretability technique that takes the residual stream at some layer and pushes it through the final unembedding layer, as though it were the finished product, and then reads its distribution over outputs. Here the output is the move policy distribution. - decodable : when the move scores highly large logits after the logit lens - policy : the network's output probability distribution over all 4352 moves. Maia plays directly from this with no search About the model MAIA 3 is a bidirectional encoder over all chess board squares: one board goes in as 64 square tokens with pieces represented by a one hot vector, and a single forward pass returns a full policy over all 4352 moves plus a win/draw/loss evaluation. There is nothing autoregressive and no sequence of tokens as input, so we can easily logit lens at each layer. Additionally, its attention heads use two schemes: a "semantic" standard Q,K self-attention, and a "geometric" GAB scheme to embed complex positional information, given that chess pieces all move in different ways which complicates simple attention patterns. My primary interest in studying this model is its skill knob which turns a classic cognitive-science question de Groot; Chase & Simon: experts don't search more, they re-represent into runnable experiments, a quintessential attraction of mechanistic interpretability work I predict that: 1 genuine tactical representations snap layers earlier in the network as conditioned skill rises, 2 features become sparser. I will be researching this immediately. Interactive App I also vibe coded a satisfying interactive app that allows you to play a game against the engine at any mimicked human skill level and view each split attention head per square, watch the residual stream evolve, logit lens each layer, and more. Repo: https://github.com/David-31415/chessformer interp https://github.com/David-31415/chessformer interp To keep the investigation clean, I decided to study what I believe is the least confounded type of fork: A knight fork of only a king and queen including bishops, for instance, will inevitably lead to internal representations of defended pieces and knight-bishop exchange values : Additionally, I required that: I mined thousands of public Lichess.org games. Position FEN: "6k1/1pq2ppp/p7/3p4/1P1Nn3/P2QPP2/1B4Pb/7K b - - 1 26". In case it is not clear, Nf2 attacks both the enemy queen and king, 'forking' the two pieces. Nf2 is also checkmate which is a confound that led me to an important finding that is addressed in part 2 I hooked the MaiaEngine class to cache the residual stream at every layer. I read it out at the 18 points that correspond to the residual stream: embed in, postattn 0n after attention n , block 0n after MLP n , encoder out . For the fork, the move seemingly starts with relatively low logits until it snaps upward at late attention readout points. We have N=1. To recover the typical trajectory I run 500 forks through the same logit lens, stack their per-depth curves into a 500x18 tensor, and summarize with the mean and standard deviation across positions. The mean curve ±1 std shows the depth at which the forking move policy typically climbs. The layers where the curve rises fastest are quite plausibly the ones doing the internal computation of assembling the tactic. The mean smooths over the exact layer at which each fork transitions from unrepresented to represented. So I record the first point at which its fork logit crosses +2.5, admittedly somewhat arbitrarily. The sharp peak at postattn 05 block 5's attention layer is exciting to see. The fact that it sits at an attention readout point makes sense computationally. Attention layers are the only place information moves between squares; MLPs compute information square by square. A fork is clearly relational since it depends on the position of two other forked pieces, so the snap should appear exactly where some attention information routing happens. It appears as though for most strong moves, block 7 amplifies the logits further as seen above. The very early outliers seem worth investigating too. See postattn 01 . The early-firing forks are those already above threshold within the first few layers, especially postattn 01 . These cases are very interesting for further study regarding, among other things, whether more or fewer early firing forks exist as the model's skill knob is dialed. One such example is shown below: At first glance of the positions in question, I hypothesize that part of the early resolution of the fork logit has to do with the fork square being very protected, which might be a simple feature that early layers represent. Here the knight is very safely guarded by a pawn, bishop, and rook. The fork logit snaps at postattn 05 , so it is worth looking at what that attention layer is comprised of. Each head's pattern is softmax qk + gab : qk is the standard self attention matching between two squares' representations semantics , while gab is a bias drawing on a library of 64×64 attention templates gab shared weight that is shared across every layer and head. Because a move's logit is proj sq from from · proj sq to to , the squares that matter are the knight's origin and its landing square. First it is worth decomposing block 5's attention into the 8 heads' individual attention scores between the Knight's square and the forking square. Again, consider the example position: In block 5, head 5, the knight's square gives 23.8% of its post-softmax attention to the landing square. Each square's attention distributes over 64 sources every row sums to 1 , so the null hypothesis uniform expected value is 1.6%: this is 15× that, and no other head comes close for this pair. Visualizing the full attention heads in classic mech interp style: With circuitsvis.attention patterns : A more natural way to view attention, below: Columns are heads, rows are qk , gab , summed attention scores where the chess board is colored differently depending on the querying square with the white outline. Attention is represented vertically flipped. Notice the shape of head 5 GAB. head 5, landing f2 - forked: d3, h1 gab: 3.045553207397461, 3.391704797744751 qk : 0.6249876022338867, 1.4472296237945557 So geometric GAB dominates here. I was originally inspired by the Monroe, et al paper: https://arxiv.org/abs/2605.19091 https://arxiv.org/abs/2605.19091 , and its beautiful transcoder-found features prompted in me questions of compositionality and superposition. I have been thinking a lot about a thesis that "Compression is necessary but not sufficient for intelligence, and insight is the name for when it happens." Insight is the constrained search through a space of representations which yields a compression. It is the mechanism by which a system becomes more intelligent like sudden non-additive chess tactic re-representations... . I really encourage people to check out the app I made and have some fun with it. Possible confounders: I had a relatively difficult time understanding the mechanics of GAB and some details about the architectural choices of Maia 3, and still am not sure I faithfully conveyed its details 100% correctly. The transcoder weights would prove very useful additions to further exploration and I am trying to obtain access to them from the CSS lab that made Maia 3. This was my first mechanistic interpretability experiment I started rapidly teaching myself the field this summer after being awestruck by some of Chris Olah's blog posts like the tanh NN visualization so I am not certain if I missed any important steps, but I tried to approach it like a math proof: what do I need to show, what would falsify this, etc. I enjoyed it very much. I am thankful to the CSS Toronto lab for creating the engine, to Professor Colin Allen and Julie Litman for helping me with the write up, and to all the wonderful interpretability papers Anthropic has released over the past few years that have gotten me so interested in the field. I am very open to feedback and comments of any nature And please visit davidlitman.com http://davidlitman.com for more work if you think we have common interests. More to come...