{"slug": "moonshot-ai-releases-kimi-k3-a-2-8-trillion-parameter-open-moe-model-with-kimi", "title": "Moonshot AI Releases Kimi K3: A 2.8 Trillion Parameter Open MoE Model With Kimi Delta Attention and 1M Context", "summary": "Moonshot AI released Kimi K3, a 2.8-trillion-parameter open Mixture-of-Experts model with Kimi Delta Attention and a 1-million-token context window, claiming it is the world's first open 3T-class model. The model introduces architectural innovations like Kimi Delta Attention and Attention Residuals to improve efficiency and performance on long-context tasks, though it still trails top proprietary models like Claude Fable 5 and GPT 5.6 Sol on some benchmarks.", "body_md": "Moonshot AI just released ** Kimi K3**. It is a 2.8-trillion-parameter model with native vision and a 1-million-token context window. Moonshot calls it the world’s first open 3T-class model.\n\n**What is Kimi K3?**\n\nKimi K3 is a sparse Mixture-of-Experts (MoE) model built on two architectural updates. Those are Kimi Delta Attention (KDA) and Attention Residuals (AttnRes). Both change how information flows across sequence length and model depth. K3 targets long-horizon coding, knowledge work, and reasoning.\n\nMoonshot team states K3 is the first open model to reach 2.8 trillion parameters. For nine of the past twelve months, Kimi models set the upper bound of open-model sizes.\n\nMoonshot is also direct about where K3 sits. Overall performance still trails the most powerful proprietary models, Claude Fable 5 and GPT 5.6 Sol. Across Moonshot’s own evaluation suite, K3 consistently outperformed other tested models.\n\n**The Architecture Underneath**\n\nKimi Delta Attention (KDA) is a hybrid linear attention mechanism. Moonshot states it enables up to 6.3x faster decoding in million-token contexts.\n\nAttnRes works along the other axis, which is depth. It selectively retrieves representations across depth rather than accumulating them uniformly. Moonshot states AttnRes delivers roughly 25% higher training efficiency at under 2% additional cost.\n\nSparsity is the third lever. K3 uses Stable LatentMoE, effectively activating 16 of 896 experts. At that sparsity, routing and optimization become first-order challenges. Quantile Balancing derives expert allocation directly from router-score quantiles. That eliminates heuristic updates and a sensitive balancing hyperparameter. Per-Head Muon extends Muon by optimizing attention heads independently. Sigmoid Tanh Unit (SiTU) and Gated MLA improve activation control and attention selectivity respectively.\n\nRefined training and data recipes accompany those structural changes. Together they yield roughly 2.5x better overall scaling efficiency than Kimi K2.\n\nThose choices carry into serving. K3 applies quantization-aware training from the SFT stage onward. It uses MXFP4 weights with MXFP8 activations for broad hardware compatibility. Moonshot team recommends supernode configurations with 64 or more accelerators. Because KDA poses new challenges for prefix caching, Moonshot contributed an implementation to vLLM.\n\n**Performance**\n\nWith the mechanics established, the published scores are easier to read. All K3 results use reasoning effort set to max. Harnesses differ per benchmark: KimiCode, Claude Code, or Codex.\n\n| Benchmark | Kimi K3 | Fable 5 (w/ fallback) | GPT 5.6 Sol | Opus 4.8 | GLM-5.2 |\n|---|---|---|---|---|---|\n| DeepSWE | 67.5 | 70.0 | 73.0 | 59.0 | 46.2 |\n| Program Bench | 77.8 | 76.8 | 77.6 | 71.9 | 63.7 |\n| Terminal Bench 2.1 | 88.3 | 84.6 | 88.8 | 84.6 | 82.7 |\n| FrontierSWE | 81.2 | 86.6 | 71.3 | 66.7 | 67.3 |\n| SWE Marathon | 42.0 | 35.0 | 39.0 | 40.0 | 13.0 |\n| BrowseComp | 91.2 | 88.0 | 90.4 | 84.3 | — |\n| Automation Bench | 30.8 | 29.1 | 29.7 | 27.2 | 12.9 |\n| GPQA-Diamond | 93.5 | 92.6 | 94.1 | 91.0 | 91.2 |\n| HLE-Full | 43.5 | 53.3 | 44.5 | 49.8 | — |\n| MMMU-Pro | 81.6 | 81.2 | 83.0 | 78.9 | — |\n| OmniDocBench | 91.1 | 89.8 | 85.8 | 87.9 | — |\n\nTwo caveats shape this table. 'With fallback' means requests Fable 5 refuses under its usage policy route to Opus 4.8. Also, BrowseComp used context compaction triggered at 300K tokens. Without that context management, K3 scores 90.4.\n\nSo K3 leads Program Bench, SWE Marathon, BrowseComp, Automation Bench, and OmniDocBench. It trails Fable 5 on FrontierSWE and HLE-Full, and GPT 5.6 Sol on DeepSWE.\n\n**Use Cases and Examples**\n\n| Use case | Reported example | Relies on |\n|---|---|---|\n| Repo-scale engineering | Long sessions, minimal human oversight | Kimi Code, `/model` |\n| Vision in the loop | Iterating between code and live screenshots | Vision, `ms://<file-id>` |\n| Research reproduction | I–Love–Q relations: 20+ papers, 3,000+ lines of Python | 1M context, auto caching |\n| Deep research reports | 42-year ASIC study: 2.8k+ fetches, 11k+ pages | Kimi Work, Widgets |\n| Document parsing | OmniDocBench score of 91.1 | Vision, structured output |\n\nMoonshot team states one native multimodal architecture handles text, images, and video together.\n\n**Access and a Minimal Call**\n\nK3 is live on Kimi.com, Kimi Work, Kimi Code, and the API. Access runs through the OpenAI SDK against a Moonshot base URL.\n\n``` python\nfrom openai import OpenAI\nimport os\n\nclient = OpenAI(api_key=os.environ[\"MOONSHOT_API_KEY\"],\n                base_url=\"https://api.moonshot.ai/v1\")\n\ncompletion = client.chat.completions.create(\n    model=\"kimi-k3\",\n    reasoning_effort=\"max\",\n    messages=[{\"role\": \"user\", \"content\": \"Introduce Kimi K3 in one sentence.\"}],\n)\nprint(completion.choices[0].message.content)\n```\n\nFour rules matter. `reasoning_effort`\n\nsupports only `max`\n\n, and the K2.x `thinking`\n\nparameter must not be used. `temperature`\n\n, `top_p`\n\n, and `n`\n\nare fixed, so omit them. `max_completion_tokens`\n\ndefaults to 131072 and reaches 1048576. In multi-turn and tool calls, return the complete assistant message.\n\nPricing is flat, with no tiering by context length. Cache-hit input is $0.30/MTok, cache-miss is $3.00/MTok, and output is $15.00/MTok. The cache-hit rate is therefore the number to watch. Moonshot team reports above 90% cache hits in coding workloads.\n\n**Key Takeaways**\n\n- Kimi K3 is a 2.8T-parameter open MoE model activating 16 of 896 experts.\n- KDA, AttnRes, sparsity, and refined recipes give ~2.5x better scaling than K2.\n- K3 leads BrowseComp, SWE Marathon, OmniDocBench; trails Fable 5 on FrontierSWE and HLE-Full.\n- OpenAI-SDK compatible at $0.30/$3.00/$15.00 per MTok, with 1M context.\n\nCheck out the ** Technical details** and\n\n**Try here****.** Also, feel free to follow us on\n\n**and don’t forget to join our**[Twitter](https://x.com/intent/follow?screen_name=marktechpost)\n\n**and Subscribe to**\n\n[150k+ML SubReddit](https://www.reddit.com/r/machinelearningnews/)**. Wait! are you on telegram?**\n\n[our Newsletter](https://www.aidevsignals.com/)\n\n[now you can join us on telegram as well.](https://t.me/machinelearningresearchnews)Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? [Connect with us](https://forms.gle/wbash1wF6efRj8G58)\n\nAsif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.", "url": "https://wpnews.pro/news/moonshot-ai-releases-kimi-k3-a-2-8-trillion-parameter-open-moe-model-with-kimi", "canonical_source": "https://www.marktechpost.com/2026/07/16/moonshot-ai-releases-kimi-k3-a-2-8-trillion-parameter-open-moe-model-with-kimi-delta-attention-and-1m-context/", "published_at": "2026-07-16 23:47:05+00:00", "updated_at": "2026-07-16 23:59:52.279530+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-products", "ai-infrastructure"], "entities": ["Moonshot AI", "Kimi K3", "Claude Fable 5", "GPT 5.6 Sol", "Opus 4.8", "GLM-5.2", "vLLM"], "alternates": {"html": "https://wpnews.pro/news/moonshot-ai-releases-kimi-k3-a-2-8-trillion-parameter-open-moe-model-with-kimi", "markdown": "https://wpnews.pro/news/moonshot-ai-releases-kimi-k3-a-2-8-trillion-parameter-open-moe-model-with-kimi.md", "text": "https://wpnews.pro/news/moonshot-ai-releases-kimi-k3-a-2-8-trillion-parameter-open-moe-model-with-kimi.txt", "jsonld": "https://wpnews.pro/news/moonshot-ai-releases-kimi-k3-a-2-8-trillion-parameter-open-moe-model-with-kimi.jsonld"}}