{"slug": "kimi-k3-architecture-notes", "title": "Kimi K3 Architecture Notes", "summary": "Kimi K3, a 2.8-trillion-parameter open-weight model from Moonshot AI, is the largest open-weight model to date, scaling up their earlier Kimi Linear architecture from 48B parameters. The architecture introduces LatentMoE for inference efficiency, attention residuals for improved residual paths, and replaces all RoPE layers with NoPE (No Positional Embeddings), making it the first frontier-level model to do so. Kimi K3 also includes native multimodal support, according to a Substack note by Sebastian Raschka.", "body_md": "# Kimi K3 Architecture Notes\n\nThe Kimi K3 architecture figure for yesterday’s big open-weight model release, along with some observations and thoughts.\n\n-\nYes, it looks relatively complicated, but it’s essentially a scaled-up production version of their\n\n[Kimi Linear model](/llm-architecture-gallery/#card-kimi-linear-48b-a3b)they released last year (scaled up from 48B -> 2.8T; K3 is by far the biggest open-weight model right now) -\nThe one new component compared to Kimi Linear is the\n\n[LatentMoE](/llm-architecture-gallery/latent-moe/). I omitted it in the figure below since it’s already very crowded, but that’s essentially the same LatentMoE as in Nemotron 3 Ultra (you can find it in my[LLM Architecture Gallery](/llm-architecture-gallery/)if you are curious). The idea here is to compress (down-project) large linear layers similar to[multi-head latent attention](/llm-architecture-gallery/mla/). -\nKimi K3’s overall trend (similar to Nemotron 3, DeepSeek V4, and others) is also towards better inference efficiency. That is, there are many components that replace existing components with efficiency-tweaked versions. I.e.,\n\n[MoE](/llm-architecture-gallery/moe/)-> LatentMoE, regular attention -> multi-head latent attention and[Kimi Delta Attention](/llm-architecture-gallery/hybrid-attention/). (I also have short tutorials and write-ups in my[gallery](/llm-architecture-gallery/)if you are curious about additional details). -\nThe one component change that is not an efficiency tweak is\n\n[attention residuals](/llm-architecture-gallery/attention-residuals/). Like DeepSeek V4 improved the residual path with mHC (manifold-constrained Hyper-Connections), attention residuals are a way to improve the residual path, but it works a bit differently. I.e., mHC made the residual path wider. Attention residuals (also already part of Kimi Linear) connect the residuals across layers; the connection itself uses an attention score for an important/contribution weight. According to the report, it improves the validation loss and downstream performance (a bit) consistently and adds about 4% in training cost and 2% in inference cost. -\nInterestingly, Kimi K3 got rid of all RoPE layers and uses\n\n[NoPE](/llm-architecture-gallery/nope/)(No Positional Embeddings) everywhere instead. (Again, this is inherited from Kimi Linear). In other architectures, the recent trend was towards RoPE in local attention layers (like[sliding window attention](/llm-architecture-gallery/swa/)) and NoPE in the global layers. There were a few architectures that only used NoPE everywhere, but this is the first frontier-level one as far as I know. -\nKimi K3 now also has native multimodal support, which is great!\n\nThere are several other interesting training tidbits in the technical report, but that’s it from the architecture front so far. A really great release overall.\n\nSource: website version of my [Substack note](https://substack.com/@rasbt/note/c-303378576).\n\n## Read Next\n\n[A Few Notable Open-Weight Models This Week Short note on the architectures of six new open-weight models, including Nanbeige 4.2, Laguna S 2.1, Motif-3-Beta, Solar Open 2, Antares 1B, and BTL-3.](/blog/2026/notable-open-weight-models-this-week.html)\n\n[Correction for Listing 6.5 in Build a Reasoning Model From Scratch Short correction note for the random seed in Listing 6.5 on page 198 of Build a Reasoning Model From Scratch.](/blog/2026/reasoning-model-listing-6-5-correction.html)\n\n[Inkling: A New Open-Weight 975B MoE with a Few Surprises Short note on Thinking Machines Lab's 975B Inkling model, including benchmarks, sparse MoE design, short convolutions, RMSNorm, and position bias.](/blog/2026/inkling-architecture-benchmark-notes.html)", "url": "https://wpnews.pro/news/kimi-k3-architecture-notes", "canonical_source": "https://sebastianraschka.com/blog/2026/kimi-k3-architecture-notes.html", "published_at": "2026-07-28 08:38:20+00:00", "updated_at": "2026-07-28 15:56:21.363543+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-products"], "entities": ["Kimi K3", "Moonshot AI", "Kimi Linear", "LatentMoE", "NoPE", "Sebastian Raschka", "Nemotron 3 Ultra", "DeepSeek V4"], "alternates": {"html": "https://wpnews.pro/news/kimi-k3-architecture-notes", "markdown": "https://wpnews.pro/news/kimi-k3-architecture-notes.md", "text": "https://wpnews.pro/news/kimi-k3-architecture-notes.txt", "jsonld": "https://wpnews.pro/news/kimi-k3-architecture-notes.jsonld"}}