Learn Linear Attention From Kimi K3's KDA Mechanism in 20 Lines of Python

A developer explains Kimi K3's KDA (Kimi Delta Attention) mechanism, a variant of linear attention that enables the model's 1-million-token context window. The post provides a 20-line Python example contrasting standard softmax attention with linear attention, showing how the latter avoids the N×N matrix computation. The author notes that while linear attention makes long contexts computationally feasible, KDA adds a delta term to recover expressiveness lost by plain linear attention.

Kimi K3 made headlines with 2.8 trillion parameters, but the most interesting part for learners is not the size. It is the attention mechanism. K3 uses something called KDA-Kimi Delta Attention. It is a variant of linear attention, which is a different way of computing the relationship between tokens in a sequence. If you are learning how transformers work, understanding the difference between standard softmax attention and linear attention will help you see why architecture choices matter more than parameter counts. In a standard transformer, every token looks at every other token. If your sequence has N tokens, the attention matrix is N x N. For a 1-million-token context window which K3 supports , that matrix would be enormous-impossibly large to compute directly. That is why most models use tricks like sliding windows or sparse patterns to avoid computing the full matrix. Linear attention rewrites the formula so you do not need the full N x N matrix. Instead of computing attention between every pair of tokens, you maintain a running state that gets updated as you process each token. The cost per token becomes roughly constant instead of growing with sequence length. This is what makes K3's 1M context window practical. This is not KDA itself-KDA adds a delta mechanism on top of linear attention that the K3 team designed to recover some of the expressiveness that linear attention loses. But this example shows the core idea of linear attention: python import numpy as np Simulate a small sequence of 8 tokens, each with 4-dimensional features np.random.seed 42 seq len = 8 dim = 4 queries = np.random.randn seq len, dim keys = np.random.randn seq len, dim values = np.random.randn seq len, dim Standard attention: N x N matrix attn scores = queries @ keys.T 8x8 matrix attn weights = np.exp attn scores / np.exp attn scores .sum axis=-1, keepdims=True standard output = attn weights @ values 8x4 Linear attention: maintain a running state, no N x N matrix state = np.zeros dim, dim linear output = for i in range seq len : state += np.outer keys i , values i update running state linear output.append queries i @ state query against state linear output = np.array linear output 8x4 print "Standard attention output shape:", standard output.shape print "Linear attention output shape:", linear output.shape print "Outputs are different expected but both are 8x4" Run this and you will see that both methods produce the same shape output, but the linear version never builds the full 8x8 matrix. For 8 tokens the difference is nothing. For 1 million tokens, it is the difference between feasible and impossible. KDA Kimi Delta Attention introduces a delta term that helps the model retain information that plain linear attention tends to blur over long sequences. The technical details are in Moonshot AI's published materials. For a beginner, the key takeaway is: linear attention makes long context cheap, and KDA tries to keep it good. If you are starting out with machine learning, do not rush to run K3. Start here: Parameters get the headlines. Architecture is where the actual engineering happens. Disclosure: I'm a MonkeyCode user sharing my own experience, not affiliated with the project. MonkeyCode is an open-source AI coding platform: https://github.com/chaitin/MonkeyCode https://github.com/chaitin/MonkeyCode