Inside Kimi K3 Technical Report

Moonshot AI's Kimi K3 technical report details a 2.8-trillion-parameter model with 104 billion activated parameters, a 1-million-token context window, and native vision, achieving 2.5x scaling efficiency over Kimi K2. The architecture introduces Kimi Delta Attention (KDA), a linear attention mechanism with diagonalized gates, and uses Multi-head Latent Attention (MLA) without position encoding, alongside an Attention Residual mechanism to support 93 layers—a 52% increase from Kimi K2's 61 layers.

Kimi-series latest model, K3, got scaled up to 2.8 trillion parameters. Impressive. Moonshot AI’s Kimi K3 technical report https://arxiv.org/pdf/2607.24653 opens with a model that is, on paper, almost three times the size of Kimi K2–2.8T total parameters, 104B activated, a 1M-token context window, and native vision. The loss comparison shows a 2.5X scaling efficiency over Kimi K2 — just another proof that the scaling law still has its room. However, the scaling law itself is just a piece of evidence on “what works” — but the underlying question, “how things work”, is the actual key in the report. In this article, I’ll walk through four architectural ideas that make Kimi K3’s scale tractable: Then close with the infrastructure co-design that had to happen alongside all of it, continuing a trend that started with DeepSeek V3 https://arxiv.org/html/2412.19437 . The core attention layer is made of 3 KDA layers + 1 Gated MLA layer, each followed by a stable latentMoE layer which we’ll talk about laer . First, we’ll explain the KDA layer. Kimi K3 doesn’t use classical multi-head attention as its default sequence mixer. Instead, three out of every four attention layers are Kimi Delta Attention KDA , a linear attention mechanism first proposed in Kimi Linear 2025 and directly descended from Gated DeltaNet . The DeltaNet idea is simple: to address the underperforming issue caused by the vanilla linear attention design, a delta rule based on generalized Householder transformation was proposed to update the linear recurrence: Then Gated DeltaNet further added a scalar forget gate to further stabilize the learning process: Which Kimi Delta Attention KDA further added a diagonalized gate to enable fine-grained control of the decay: To improve computational efficiency, KDA could be further rewritten into a chunk-wise parallel format https://arxiv.org/pdf/2510.26692 which is a typical format for linear attention computation . To address the precision overflow issue, a negative-softplus mapping is introduced to bound the decay logits. The Multi-head Latent Attention MLA is a mechanism DeepSeek-V2 https://arxiv.org/abs/2405.04434 introduced to shrink the KV cache by compressing keys and values into a shared low-rank latent before reconstructing per-head projections at attention time. While classical MLA applies RoPE to inject positional information, Kimi K3 drops this entirely — its MLA layers use NoPE No Position Encoding . The reasoning is architectural: the KDA layers already provide position-sensitive, recency-aware mixing through their recurrent decay, so the MLA layers are freed up to do what they’re best at. It also sidesteps a very practical long-context headache: no RoPE frequency base to retune, no YaRN interpolation needed when extending context length , since there’s no rotary embedding to extrapolate in the first place. On top of this, K3 adds a full-rank, input-dependent output gate to MLA mirroring a similar gate added to KDA , letting each token modulate which channels it reads from global attention. This is probably one of the most interesting concepts proposed by the technical report: similar to how transformers addressed sequential dependencies with attention, the Attention Residual attempts to formulate the residuals in attention format: When moving into Kimi K3 architecture, it means calculating the residual per-pair of layers: Given the model depth < 100, the full computation cost is O L²d , which is still affordable, and can be reduced if the residual is computed in a block-wise style. So why adding residuals in such a layer-wise style? The reason is simple — Kimi K3 scaled the number of layers up to 93, comparing Kimi K2 which only has 61 layers. The 52% incease of model layers causes a huge bottleneck on gradient backpropagation — which cannot be resolved by classical skip connection, and needs more dense representation. Kimi K3 pushes the number of experts in its MoE architecture much further — 896 routed experts with 16 activated per token versus K2’s 384 routed / 8 active — and that 133% extra jump in sparsity breaks two things that used to work fine at smaller scale. The architectural fix is LatentMoE : shared experts still operate on the model’s full hidden width, but routed experts operate in a much narrower latent space, reached via a down-projection before dispatch and an up-projection after aggregation. This decouples the router’s cost from the full model width, which is what makes activating 16 of 896 experts per token affordable at all. The stability problems remain, though. Kimi K3 addresses this two approaches: i with an RMSNorm before the up-projection + a new activation function, SiTU-GLU Sigmoid Tanh Unit GLU , which soft-caps both branches of a SwiGLU-style gate with a scaled tanh — keeping the near-origin behaviour that makes SwiGLU work well, while bounding the output so large-magnitude coordinates can’t blow up in low precision; ii Quantile Balancing : directly sets the expert bias to the score-quantile that would deliver that expert its target load, estimated per training step from a histogram of routing margins to avoid gathering millions of individual values for an exact quantile . One of the more surprising empirical results in the report has nothing to do with attention or MoE. Kimi K2.5’s vision encoder, like most multimodal LLM vision towers, was initialized from a contrastively pre-trained model SigLIP , on the assumption that contrastive pre-training gives the encoder useful visual priors before it’s ever attached to a language model. Kimi K3’s vision encoder, MoonViT-V2 , is trained entirely from scratch using next-token prediction — no contrastive stage at all. The stated motivation is training stability: when a SigLIP-initialized encoder is jointly optimized with the LLM, gradient norms spike persistently, while the from-scratch encoder trains smoothly throughout. The more interesting finding is the downstream one — MoonViT-V2 matches the SigLIP-initialized baseline across vision benchmarks, which the authors read as evidence that contrastive pre-training isn’t actually a necessary ingredient for a strong multimodal LLM vision encoder, at least at this scale and with enough NTP-style multimodal data. Here’s the pattern that ties the whole report together, and it’s the same one DeepSeek V3 first made explicit: at frontier scale, infrastructure isn’t a neutral platform you deploy a model onto — it has to be co-designed with the specific bottlenecks the architecture creates. DeepSeek V3 https://arxiv.org/html/2412.19437 ’s DualPipe was built to solve a communication overhead problem created by its own MoE design. Kimi K3 does the same thing in several places at once: Kimi K3 is not only a model that scaled up. The scaling effort itself is a combination of a series of research results together with infra practices, which made the scaling possible. It also proved a few things: Inside Kimi K3 Technical Report https://pub.towardsai.net/inside-kimi-k3-technical-report-aa7a9e671de7 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.