Kimi K3 made headlines with 2.8 trillion parameters, but the most interesting part for learners is not the size. It is the attention mechanism.
K3 uses something called KDA-Kimi Delta Attention. It is a variant of linear attention, which is a different way of computing the relationship between tokens in a sequence. If you are learning how transformers work, understanding the difference between standard softmax attention and linear attention will help you see why architecture choices matter more than parameter counts.
In a standard transformer, every token looks at every other token. If your sequence has N tokens, the attention matrix is N x N. For a 1-million-token context window (which K3 supports), that matrix would be enormous-impossibly large to compute directly.
That is why most models use tricks like sliding windows or sparse patterns to avoid computing the full matrix.
Linear attention rewrites the formula so you do not need the full N x N matrix. Instead of computing attention between every pair of tokens, you maintain a running state that gets updated as you process each token. The cost per token becomes roughly constant instead of growing with sequence length.
This is what makes K3's 1M context window practical.
This is not KDA itself-KDA adds a delta mechanism on top of linear attention that the K3 team designed to recover some of the expressiveness that linear attention loses. But this example shows the core idea of linear attention:
import numpy as np
np.random.seed(42)
seq_len = 8
dim = 4
queries = np.random.randn(seq_len, dim)
keys = np.random.randn(seq_len, dim)
values = np.random.randn(seq_len, dim)
attn_scores = queries @ keys.T # 8x8 matrix
attn_weights = np.exp(attn_scores) / np.exp(attn_scores).sum(axis=-1, keepdims=True)
standard_output = attn_weights @ values # 8x4
state = np.zeros((dim, dim))
linear_output = []
for i in range(seq_len):
state += np.outer(keys[i], values[i]) # update running state
linear_output.append(queries[i] @ state) # query against state
linear_output = np.array(linear_output) # 8x4
print("Standard attention output shape:", standard_output.shape)
print("Linear attention output shape:", linear_output.shape)
print("Outputs are different (expected) but both are 8x4")
Run this and you will see that both methods produce the same shape output, but the linear version never builds the full 8x8 matrix. For 8 tokens the difference is nothing. For 1 million tokens, it is the difference between feasible and impossible.
KDA (Kimi Delta Attention) introduces a delta term that helps the model retain information that plain linear attention tends to blur over long sequences. The technical details are in Moonshot AI's published materials. For a beginner, the key takeaway is: linear attention makes long context cheap, and KDA tries to keep it good.
If you are starting out with machine learning, do not rush to run K3. Start here:
Parameters get the headlines. Architecture is where the actual engineering happens.
Disclosure: I'm a MonkeyCode user sharing my own experience, not affiliated with the project. MonkeyCode is an open-source AI coding platform: https://github.com/chaitin/MonkeyCode