{"slug": "beyond-bigger-moe-how-kimi-k3-scales-context-depth-and-agents", "title": "Beyond Bigger MoE: How Kimi K3 Scales Context, Depth, and Agents", "summary": "Moonshot AI's Kimi K3 model scales to 2.8T total parameters with 104B activated per token, using hybrid attention, Attention Residuals, and Stable LatentMoE to support 1M-token agentic trajectories. The model interleaves three Kimi Delta Attention layers with one global Gated MLA layer, employs no positional encoding, and activates 16 of 896 routed experts per token. Kimi K3 also integrates a 401M-parameter MoonViT-V2 vision encoder trained from scratch with next-token prediction.", "body_md": "## Beyond Bigger MoE: How Kimi K3 Scales Context, Depth, and Agents\n\nRecent frontier-model development has increasingly emphasized reinforcement learning and test-time computation: make a strong pretrained model reason longer, use more tools, and execute increasingly complex trajectories. **Kimi K3** argues that post-training alone is not enough. Moonshot AI scales both sides simultaneously: the pretrained foundation grows to **2.8T total** parameters with **104B activated** per token, while post-training expands to **1M-token** agentic trajectories, multiple reasoning-effort levels, coding, knowledge work, and general tool use.\n\nKimi K3 organizes scaling around three kinds of information flow:\n\n- Across sequence length, it interleaves three efficient\n**Kimi Delta** layers with one global Gated[Attention](https://dswok.com/Deep-Learning/Attention)**MLA** layer. - Across depth,\n**Attention Residuals** let layers selectively retrieve representations from earlier blocks rather than receiving all previous computation through a single accumulated residual stream. - Across width,\n**Stable LatentMoE** expands the model to 896 routed experts while activating 16 per token through a lower-dimensional expert pathway.\n\n### Hybrid attention without positional encoding\n\n## Hybrid attention: local recurrence with periodic global access\n\nFull global attention becomes increasingly expensive as context grows, while purely recurrent linear attention compresses the entire history into a fixed-size state and can lose information. Kimi K3 combines both approaches: its backbone repeats three Kimi Delta Attention layers followed by one global Gated MLA layer, with an additional MLA layer at the end.\n\nK3 interleaves **3 KDA layers per 1 Gated MLA layer**. Kimi Delta Attention, introduced in Kimi Linear, is a linear-attention delta-rule recurrence with channel-wise forget gates: instead of a KV cache that grows with sequence length, each layer maintains a fixed-size state updated by a delta rule, and per-channel gates control how quickly old information decays. A new addition to it is a **lower-bounded scaled-sigmoid decay**, which keeps decay values away from the extremes so that all chunkwise computation tiles run on Tensor Cores. The periodic Gated MLA layers preserve exact global attention where the recurrence would lose detail, the same reliability role that full-attention layers play in other hybrid stacks.\n\nThere is no explicit positional encoding (**NoPE**). Position emerges implicitly from KDA’s gating and decay, and the practical payoff comes at long context: with no RoPE, there are no frequencies to interpolate or rescale when the window is extended to 1M tokens.\n\n### Attention Residuals and Stable LatentMoE\n\n**Attention Residuals** replace the single accumulated residual stream. Instead of every layer adding its output into one shared vector, each layer forms a learned pseudo-query and softmax-attends over the embedding and all preceding layers’ outputs, choosing what to read from earlier depth. K3 uses a block variant, 8 blocks of 12 layers, so the memory cost of retaining earlier outputs stays bounded. The intent is similar to Hyper-Connections: widen the path along depth so later layers can access earlier computation selectively rather than through one lossy sum.\n\nThe feed-forward layers use **Stable LatentMoE**: tokens are projected into a compact latent space at half the hidden dimension and routed to **896 experts with 16 active per token**, plus 2 full-width shared experts.\n\nSuch sparsity is often unstable and requires several things to be stable: an RMSNorm before the up-projection, a bounded **SiTU-GLU** activation replacing SwiGLU, and **Quantile Balancing**, which sets each expert’s routing bias directly from router-score quantiles.\n\nVision is native from the start of pre-training through a 401M-parameter **MoonViT-V2** encoder trained entirely from scratch with next-token prediction: the authors abandoned [SigLIP](https://andlukyane.com/blog/paper-review-siglip2) initialization after the pretrained encoder caused persistent gradient spikes during joint training, and the from-scratch encoder matches it on vision evals.\n\n### Pre-training\n\nThe corpus spans web text, code, mathematics, and knowledge data, with an addition of a large vision corpus: captions, interleaved image-text, OCR, video, and “programmatic multimodal” data that pairs code with the SVG, 3D, webpage, and CAD visuals it renders. The pipelines build on K2 and K2.5, including K2’s rephrasing recipe for knowledge and math data, and language and vision are optimized jointly from the first token, continuing the native-multimodal approach from [Kimi K2.5](https://andlukyane.com/blog/paper-review-kimik25).\n\nThe optimizer is **Per-Head Muon** with weight clipping. The claimed **~2.5x scaling-efficiency gain over K2** comes from multiple improvements: the authors re-ran scaling-law searches for batch size, learning rate, tokens-per-parameter, and model shape, and picked cosine LR decay over WSD only after running independent scaling-law searches for each schedule. Context length follows a curriculum, 8K to 64K during pre-training and 256K to 1M during cooldown.\n\n### Post-training: nine experts, one model\n\nPost-training starts with SFT on trajectories synthesized by prior domain-specialized Kimi models, then moves to RL. The authors train **9 expert policies**, a 3x3 grid of domains (general, general agents, coding agents) and reasoning-effort levels (low, high, max), each in white-box agentic environments. Rollouts use a partial-rollout scheme with per-token regularization against staleness, non-verifiable tasks are scored by an **Agentic Generative Reward Model** under a mandatory rubric protocol, and per-problem token budgets teach each effort level its compute envelope.\n\nThe experts are then consolidated into a single model via **Multi-Teacher On-Policy Distillation (MOPD)**: the student generates its own rollouts and receives a per-token clipped log-ratio reward from whichever teacher matches the task’s domain and effort level. This is the same compositional pattern [DeepSeek-V4](https://andlukyane.com/blog/paper-review-deepseekv4) adopted with its on-policy distillation of domain specialists, extended to a second axis (reasoning effort) and to agentic environments.\n\nQuantization is part of training: from SFT onward the model runs quantization-aware training with **MXFP4 expert weights** and MXFP8 activations, and the same scheme is used in RL rollouts and serving, so deployment sees the same numerics as training. The multi-token-prediction layer is fine-tuned into an EAGLE-3-style draft model for speculative decoding.\n\n### Experiments\n\nAt max effort, K3 is just behind Claude Fable 5 and GPT-5.6 Sol and ahead of everything else evaluated, including Claude Opus 4.8, GPT-5.5, and GLM-5.2. On knowledge, it is competitive at the top, with **93.5 on GPQA Diamond**, tying GPT-5.5 and just behind GPT-5.6 Sol. Research-level reasoning is a gap: on CritPt it scores **23.4 against GPT-5.6 Sol’s 32.3**\n\nOne of the more interesting parts of the report is the case-study section. In a kernel-optimization environment K3 matched Claude Fable 5 and cut its own AttnRes kernel latency by more than half; it also built **MiniTriton**, a Triton-like compiler that reaches about 90% of machine roof on an L20 matmul and trains a GPT end-to-end, and designed an INT4 inference-chip prototype (**nano-kpu**) in a 48-hour autonomous run. The report also includes a cybersecurity evaluation: the model found **16 previously unknown vulnerabilities** across six open-source projects, including two Linux kernel bugs.\n\n### Conclusions\n\nFor the past year, open-weight frontier releases have converged on roughly the same size while competing on efficiency:\n\n- DeepSeek V4 at 1.6T total parameters with hybrid compressed attention\n- Kimi K2.5 at 1T with native multimodality and Agent Swarm\n- GLM and MiniMax around or below the trillion mark with their own efficient-attention and agentic-RL recipes.\n\nThe shared assumption was that the next gains would come from test-time compute and cheaper long-context inference, not from a bigger base model. K3 use a different approach: it combines hybrid attention across tokens, learned retrieval across layers, extremely sparse latent experts, native multimodality, million-token context, and harness-diverse agentic reinforcement learning in one 2.8-trillion-parameter system.\n\nTo me the most consequential result is the demonstration that a linear-attention hybrid works at frontier scale. I also like how honest the evaluation section is. The paper states openly that it trails Claude Fable 5 and GPT-5.6 Sol, and the third-party leaderboards confirm the placement.\n\n[paperreview](/tag/paperreview)\n\n[deeplearning](/tag/deeplearning)\n\n[llm](/tag/llm)\n\n[vlm](/tag/vlm)\n\n[mllm](/tag/mllm)\n\n[rl](/tag/rl)", "url": "https://wpnews.pro/news/beyond-bigger-moe-how-kimi-k3-scales-context-depth-and-agents", "canonical_source": "https://andlukyane.com/blog/paper-review-kimik3", "published_at": "2026-08-03 00:00:00+00:00", "updated_at": "2026-08-03 10:57:02.762909+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-infrastructure"], "entities": ["Moonshot AI", "Kimi K3", "Kimi Delta Attention", "Gated MLA", "Stable LatentMoE", "MoonViT-V2", "Attention Residuals"], "alternates": {"html": "https://wpnews.pro/news/beyond-bigger-moe-how-kimi-k3-scales-context-depth-and-agents", "markdown": "https://wpnews.pro/news/beyond-bigger-moe-how-kimi-k3-scales-context-depth-and-agents.md", "text": "https://wpnews.pro/news/beyond-bigger-moe-how-kimi-k3-scales-context-depth-and-agents.txt", "jsonld": "https://wpnews.pro/news/beyond-bigger-moe-how-kimi-k3-scales-context-depth-and-agents.jsonld"}}