Beyond Bigger MoE: How Kimi K3 Scales Context, Depth, and Agents

Moonshot AI's Kimi K3 model scales to 2.8T total parameters with 104B activated per token, using hybrid attention, Attention Residuals, and Stable LatentMoE to support 1M-token agentic trajectories. The model interleaves three Kimi Delta Attention layers with one global Gated MLA layer, employs no positional encoding, and activates 16 of 896 routed experts per token. Kimi K3 also integrates a 401M-parameter MoonViT-V2 vision encoder trained from scratch with next-token prediction.

Beyond Bigger MoE: How Kimi K3 Scales Context, Depth, and Agents Recent frontier-model development has increasingly emphasized reinforcement learning and test-time computation: make a strong pretrained model reason longer, use more tools, and execute increasingly complex trajectories. Kimi K3 argues that post-training alone is not enough. Moonshot AI scales both sides simultaneously: the pretrained foundation grows to 2.8T total parameters with 104B activated per token, while post-training expands to 1M-token agentic trajectories, multiple reasoning-effort levels, coding, knowledge work, and general tool use. Kimi K3 organizes scaling around three kinds of information flow: - Across sequence length, it interleaves three efficient Kimi Delta layers with one global Gated Attention https://dswok.com/Deep-Learning/Attention MLA layer. - Across depth, Attention Residuals let layers selectively retrieve representations from earlier blocks rather than receiving all previous computation through a single accumulated residual stream. - Across width, Stable LatentMoE expands the model to 896 routed experts while activating 16 per token through a lower-dimensional expert pathway. Hybrid attention without positional encoding Hybrid attention: local recurrence with periodic global access Full global attention becomes increasingly expensive as context grows, while purely recurrent linear attention compresses the entire history into a fixed-size state and can lose information. Kimi K3 combines both approaches: its backbone repeats three Kimi Delta Attention layers followed by one global Gated MLA layer, with an additional MLA layer at the end. K3 interleaves 3 KDA layers per 1 Gated MLA layer . Kimi Delta Attention, introduced in Kimi Linear, is a linear-attention delta-rule recurrence with channel-wise forget gates: instead of a KV cache that grows with sequence length, each layer maintains a fixed-size state updated by a delta rule, and per-channel gates control how quickly old information decays. A new addition to it is a lower-bounded scaled-sigmoid decay , which keeps decay values away from the extremes so that all chunkwise computation tiles run on Tensor Cores. The periodic Gated MLA layers preserve exact global attention where the recurrence would lose detail, the same reliability role that full-attention layers play in other hybrid stacks. There is no explicit positional encoding NoPE . Position emerges implicitly from KDA’s gating and decay, and the practical payoff comes at long context: with no RoPE, there are no frequencies to interpolate or rescale when the window is extended to 1M tokens. Attention Residuals and Stable LatentMoE Attention Residuals replace the single accumulated residual stream. Instead of every layer adding its output into one shared vector, each layer forms a learned pseudo-query and softmax-attends over the embedding and all preceding layers’ outputs, choosing what to read from earlier depth. K3 uses a block variant, 8 blocks of 12 layers, so the memory cost of retaining earlier outputs stays bounded. The intent is similar to Hyper-Connections: widen the path along depth so later layers can access earlier computation selectively rather than through one lossy sum. The feed-forward layers use Stable LatentMoE : tokens are projected into a compact latent space at half the hidden dimension and routed to 896 experts with 16 active per token , plus 2 full-width shared experts. Such sparsity is often unstable and requires several things to be stable: an RMSNorm before the up-projection, a bounded SiTU-GLU activation replacing SwiGLU, and Quantile Balancing , which sets each expert’s routing bias directly from router-score quantiles. Vision is native from the start of pre-training through a 401M-parameter MoonViT-V2 encoder trained entirely from scratch with next-token prediction: the authors abandoned SigLIP https://andlukyane.com/blog/paper-review-siglip2 initialization after the pretrained encoder caused persistent gradient spikes during joint training, and the from-scratch encoder matches it on vision evals. Pre-training The corpus spans web text, code, mathematics, and knowledge data, with an addition of a large vision corpus: captions, interleaved image-text, OCR, video, and “programmatic multimodal” data that pairs code with the SVG, 3D, webpage, and CAD visuals it renders. The pipelines build on K2 and K2.5, including K2’s rephrasing recipe for knowledge and math data, and language and vision are optimized jointly from the first token, continuing the native-multimodal approach from Kimi K2.5 https://andlukyane.com/blog/paper-review-kimik25 . The optimizer is Per-Head Muon with weight clipping. The claimed ~2.5x scaling-efficiency gain over K2 comes from multiple improvements: the authors re-ran scaling-law searches for batch size, learning rate, tokens-per-parameter, and model shape, and picked cosine LR decay over WSD only after running independent scaling-law searches for each schedule. Context length follows a curriculum, 8K to 64K during pre-training and 256K to 1M during cooldown. Post-training: nine experts, one model Post-training starts with SFT on trajectories synthesized by prior domain-specialized Kimi models, then moves to RL. The authors train 9 expert policies , a 3x3 grid of domains general, general agents, coding agents and reasoning-effort levels low, high, max , each in white-box agentic environments. Rollouts use a partial-rollout scheme with per-token regularization against staleness, non-verifiable tasks are scored by an Agentic Generative Reward Model under a mandatory rubric protocol, and per-problem token budgets teach each effort level its compute envelope. The experts are then consolidated into a single model via Multi-Teacher On-Policy Distillation MOPD : the student generates its own rollouts and receives a per-token clipped log-ratio reward from whichever teacher matches the task’s domain and effort level. This is the same compositional pattern DeepSeek-V4 https://andlukyane.com/blog/paper-review-deepseekv4 adopted with its on-policy distillation of domain specialists, extended to a second axis reasoning effort and to agentic environments. Quantization is part of training: from SFT onward the model runs quantization-aware training with MXFP4 expert weights and MXFP8 activations, and the same scheme is used in RL rollouts and serving, so deployment sees the same numerics as training. The multi-token-prediction layer is fine-tuned into an EAGLE-3-style draft model for speculative decoding. Experiments At max effort, K3 is just behind Claude Fable 5 and GPT-5.6 Sol and ahead of everything else evaluated, including Claude Opus 4.8, GPT-5.5, and GLM-5.2. On knowledge, it is competitive at the top, with 93.5 on GPQA Diamond , tying GPT-5.5 and just behind GPT-5.6 Sol. Research-level reasoning is a gap: on CritPt it scores 23.4 against GPT-5.6 Sol’s 32.3 One of the more interesting parts of the report is the case-study section. In a kernel-optimization environment K3 matched Claude Fable 5 and cut its own AttnRes kernel latency by more than half; it also built MiniTriton , a Triton-like compiler that reaches about 90% of machine roof on an L20 matmul and trains a GPT end-to-end, and designed an INT4 inference-chip prototype nano-kpu in a 48-hour autonomous run. The report also includes a cybersecurity evaluation: the model found 16 previously unknown vulnerabilities across six open-source projects, including two Linux kernel bugs. Conclusions For the past year, open-weight frontier releases have converged on roughly the same size while competing on efficiency: - DeepSeek V4 at 1.6T total parameters with hybrid compressed attention - Kimi K2.5 at 1T with native multimodality and Agent Swarm - GLM and MiniMax around or below the trillion mark with their own efficient-attention and agentic-RL recipes. The shared assumption was that the next gains would come from test-time compute and cheaper long-context inference, not from a bigger base model. K3 use a different approach: it combines hybrid attention across tokens, learned retrieval across layers, extremely sparse latent experts, native multimodality, million-token context, and harness-diverse agentic reinforcement learning in one 2.8-trillion-parameter system. To me the most consequential result is the demonstration that a linear-attention hybrid works at frontier scale. I also like how honest the evaluation section is. The paper states openly that it trails Claude Fable 5 and GPT-5.6 Sol, and the third-party leaderboards confirm the placement. paperreview /tag/paperreview deeplearning /tag/deeplearning llm /tag/llm vlm /tag/vlm mllm /tag/mllm rl /tag/rl