{"slug": "when-does-a-chess-transformer-see-a-knight-fork-an-initial-result-from-logit-and", "title": "When does a chess transformer “see” a knight fork? An initial result from logit lens and attention patterns", "summary": "Researchers at the Mechanistic Interpretability project have identified that the knight-fork policy logit in the Maia 3 chess transformer snaps into place after block 5's attention layer, using logit lens and attention pattern analysis. This correlational evidence marks a step toward understanding how concrete chess tactics are represented in transformer internals, with implications for AI safety and cognitive science.", "body_md": "(parts 2 and 3 to follow)\n\nThis post is on the results of a mechanistic interpretability project aimed at understanding the internals of Maia 3: a transformer based chess bot trained to imitate human play at a chosen skill level, rather than to play optimally. My aim is to locate and causally describe one chess tactic in the Maia 3 engine. In this part 1, I describe strong correlational evidence that the** knight-fork policy logit snaps into place after block 5’s attention layer**. (Controls rule out that this is merely where the best move snaps into place at the start of part 2.) In part 2, I will further demonstrate causality and compositionality of the feature, and in part 3 describe unexpected results that shake up the pure \"knight fork attention head” thesis into something stranger and more interesting.\n\nChess is relevant to many facets of human cognition, from [chunking](https://www.youtube.com/watch?v=2pd6xtEAaJE) to sudden insight, and the hope is that general interpretability work on chess models has relevance to cognitive neuroscientific research in the future.\n\nAlong with this, the decomposable structure of the transformer architecture (see: \"A Mathematical Framework for Transformer Circuits”) adds to the intrigue of the interpretability work. Determining where in the network a concrete skill exists tells one about how its compressed representations form. A knight fork (a single knight move that attacks two valuable pieces at once) is a priori a strong target skill to analyze because it is discrete and unambiguous and important. I wanted to make headway into the question: \"at what depth does the network's internals first make a knight fork decodable?\"\n\n**Brief Notes on Mechanistic Interpretability**\n\nAs the field is relatively new, it is worth spending a few sentences to give extra context to unfamiliar readers. The aim of mechanistic interpretability is to understand AI better by reverse-engineering models into human-describable parts. The ultimate hope is that this will help safety and alignment, for example by allowing us to recognize deceptive LLMs.\n\n**Terminology I will keep constant throughout:**\n\n- **block 0–7**: the eight transformer blocks; each contains an **attention layer** and an **MLP layer**\n\n- **residual stream**: a vector; the transformer's shared communication layer to which attention and MLP layers additively read and write into, allowing for meaningful decomposition by specific layers' additions.\n\n- **readout point**: the content of the residual stream at some depth\n\n- **logit lens:** an interpretability technique that takes the residual stream at some layer and pushes it through the final unembedding layer, as though it were the finished product, and then reads its distribution over outputs. Here the output is the move policy distribution.\n\n**- decodable**: when the move scores highly (large logits) after the logit lens\n\n**- policy**: the network's output probability distribution over all 4352 moves. Maia plays directly from this with no search\n\n**About the model**\n\nMAIA 3 is a bidirectional encoder over all chess board squares: one board goes in as 64 square tokens with pieces represented by a one hot vector, and a single forward pass returns a full policy over all 4352 moves plus a win/draw/loss evaluation. There is nothing autoregressive and no sequence of tokens as input, so we can easily logit lens at each layer. Additionally, its attention heads use two schemes: a \"semantic\" standard Q,K self-attention, and a \"geometric\" GAB scheme to embed complex positional information, given that chess pieces all move in different ways which complicates simple attention patterns.\n\n**My primary interest in studying this model is its skill knob **which turns a classic cognitive-science question (de Groot; Chase & Simon: experts don't search more, they re-represent) into runnable experiments, a quintessential attraction of mechanistic interpretability work! I predict that: **1)** genuine tactical representations snap layers *earlier* in the network as conditioned skill rises, **2)** features become sparser. I will be researching this immediately.\n\n**Interactive App** I also vibe coded a satisfying interactive app that allows you to play a game against the engine at any mimicked human skill level and view each split attention head per square, watch the residual stream evolve, logit lens each layer, and more.\n\nRepo: [https://github.com/David-31415/chessformer_interp](https://github.com/David-31415/chessformer_interp)\n\nTo keep the investigation clean, I decided to study what I believe is the least confounded type of fork:\n\nA knight fork of only a king and queen (including bishops, for instance, will inevitably lead to internal representations of defended pieces and knight-bishop exchange values):\n\nAdditionally, I required that:\n\nI mined thousands of public Lichess.org games.\n\nPosition FEN: \"6k1/1pq2ppp/p7/3p4/1P1Nn3/P2QPP2/1B4Pb/7K b - - 1 26\".\n\nIn case it is not clear, **Nf2#** attacks both the enemy queen and king, 'forking' the two pieces.\n\n(**Nf2** is also checkmate which is a confound that led me to an important finding that is addressed in part 2)\n\nI hooked the MaiaEngine class to cache the residual stream at every layer. I read it out at the 18 points that correspond to the residual stream: **embed_in, postattn_0n** (after attention n), **block_0n** (after MLP n), **encoder_out**.\n\nFor the fork, the move seemingly starts with relatively low logits until it snaps upward at late attention readout points.\n\nWe have N=1. To recover the typical trajectory I run 500 forks through the same logit lens, stack their per-depth curves into a 500x18 tensor, and summarize with the mean and standard deviation across positions. The mean curve (±1 std) shows the depth at which the forking move policy typically climbs. The layers where the curve rises fastest are quite plausibly the ones doing the internal computation of assembling the tactic.\n\nThe mean smooths over the exact layer at which each fork transitions from unrepresented to represented. So I record the first point at which its fork logit crosses +2.5, admittedly somewhat arbitrarily.\n\nThe sharp peak at **postattn_05** (block 5's attention layer) is exciting to see. The fact that it sits at an attention readout point makes sense computationally. Attention layers are the only place information moves between squares; MLPs compute information square by square. A fork is clearly relational since it depends on the position of two other forked pieces, so the snap should appear exactly where some attention information routing happens. It appears as though for most strong moves, **block 7 **amplifies the logits further as seen above.\n\nThe very early outliers seem worth investigating too. See **postattn_01**.\n\nThe early-firing forks are those already above threshold within the first few layers, especially **postattn_01**. These cases are very interesting for further study regarding, among other things, whether more or fewer early firing forks exist as the model's skill knob is dialed.\n\nOne such example is shown below:\n\nAt first glance of the positions in question, I hypothesize that part of the early resolution of the fork logit has to do with the fork square being very protected, which might be a simple feature that early layers represent.\n\nHere the knight is very safely guarded by a pawn, bishop, and rook.\n\nThe fork logit snaps at **postattn_05**, so it is worth looking at what that attention layer is comprised of. Each head's pattern is softmax(qk + gab): qk is the standard self attention matching between two squares' representations (semantics), while gab is a bias drawing on a library of 64×64 attention templates (gab_shared_weight) that is shared across every layer and head.\n\nBecause a move's logit is **proj_sq_from[from] · proj_sq_to[to]**, the squares that matter are the knight's origin and its landing square.\n\nFirst it is worth decomposing block 5's attention into the 8 heads' individual attention scores between the Knight's square and the forking square. Again, consider the example position:\n\nIn block 5, head 5, the knight's square gives **23.8%** of its post-softmax attention to the landing square. Each square's attention distributes over 64 sources (every row sums to 1), so the null hypothesis (uniform) expected value is 1.6%: this is 15× that, and no other head comes close for this pair.\n\nVisualizing the full attention heads in classic mech interp style:\n\nWith circuitsvis.attention_patterns():\n\nA more natural way to view attention, below:\n\nColumns are heads, rows are (qk), (gab), (summed) attention scores where the chess board is colored differently depending on the querying square with the white outline. Attention is represented vertically flipped.\n\nNotice the shape of head 5 GAB.\n\nhead 5, landing f2 -> forked: d3, h1\n\ngab: [3.045553207397461, 3.391704797744751]\n\nqk : [0.6249876022338867, 1.4472296237945557]\n\nSo geometric GAB dominates here.\n\nI was originally inspired by the Monroe, et al paper: [https://arxiv.org/abs/2605.19091](https://arxiv.org/abs/2605.19091), and its beautiful transcoder-found features prompted in me questions of compositionality and superposition.\n\nI have been thinking a lot about a thesis that \"Compression is necessary but not sufficient for intelligence, and insight is the name for when it happens.\" Insight is the constrained search through a space of representations which yields a compression. It is the mechanism by which a system becomes more intelligent (like sudden non-additive chess tactic re-representations...).\n\nI really encourage people to check out the app I made and have some fun with it.\n\n**Possible confounders:**\n\nI had a relatively difficult time understanding the mechanics of GAB and some details about the architectural choices of Maia 3, and still am not sure I faithfully conveyed its details 100% correctly.\n\nThe transcoder weights would prove very useful additions to further exploration and I am trying to obtain access to them from the CSS lab that made Maia 3.\n\nThis was my first mechanistic interpretability experiment (I started rapidly teaching myself the field this summer after being awestruck by some of Chris Olah's blog posts like the tanh NN visualization!) so I am not certain if I missed any important steps, but I tried to approach it like a math proof: what do I need to show, what would falsify this, etc. I enjoyed it very much.\n\nI am thankful to the CSS Toronto lab for creating the engine, to Professor Colin Allen and Julie Litman for helping me with the write up, and to all the wonderful interpretability papers Anthropic has released over the past few years that have gotten me so interested in the field.\n\nI am very open to feedback and comments of any nature!\n\nAnd please visit [davidlitman.com](http://davidlitman.com) for more work if you think we have common interests.\n\nMore to come...", "url": "https://wpnews.pro/news/when-does-a-chess-transformer-see-a-knight-fork-an-initial-result-from-logit-and", "canonical_source": "https://www.lesswrong.com/posts/z7XuJWYxwiZw9Wk2W/when-does-a-chess-transformer-see-a-knight-fork-an-initial", "published_at": "2026-07-07 04:37:14+00:00", "updated_at": "2026-07-07 05:08:13.922871+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "ai-safety"], "entities": ["Maia 3", "Mechanistic Interpretability project", "de Groot", "Chase & Simon"], "alternates": {"html": "https://wpnews.pro/news/when-does-a-chess-transformer-see-a-knight-fork-an-initial-result-from-logit-and", "markdown": "https://wpnews.pro/news/when-does-a-chess-transformer-see-a-knight-fork-an-initial-result-from-logit-and.md", "text": "https://wpnews.pro/news/when-does-a-chess-transformer-see-a-knight-fork-an-initial-result-from-logit-and.txt", "jsonld": "https://wpnews.pro/news/when-does-a-chess-transformer-see-a-knight-fork-an-initial-result-from-logit-and.jsonld"}}