Inside Kimi K3: How Moonshot AI Built the Largest Open-Source Model

Moonshot AI released Kimi K3, a 2.8-trillion-parameter open-weight model, the largest open-source model as of mid-2026 and the first to reach the 3-trillion-parameter class. The model uses Kimi Delta Attention (KDA), a mechanism that reduces memory use by about 75% compared to standard attention, enabling a 1-million-token context window and multimodal support for text, images, and video.

Moonshot AI recently released Kimi K3, the largest open-weight model available as of mid-2026, and the first open model to reach the 3-trillion-parameter class. While most labs chase scale by simply adding more hardware, Moonshot took a different path. Instead of just building a bigger model, they redesigned how the model remembers things. That redesign is really the whole story of K3. The story starts with Kimi K1.5 , which focused on scaling reinforcement learning and improving the model’s basic reasoning ability. By mid-2025, the team released Kimi K2 , a roughly 1-trillion-parameter model that pushed further on architecture and training pipelines. Not long after, Kimi K2.5 added native multimodal support and stronger agentic skills, meaning the ability to use tools and carry out multi-step tasks with less human guidance. As these models grew, they ran into a bottleneck that every large language model eventually hits: memory. Standard transformer models remember everything. Every time the model generates a new word, it looks back at every previous word in the conversation. This lookup relies on something called the KV cache, which stores a representation of every token seen so far. The problem is that this cache grows with the conversation. Storage needed for it scales linearly with context length, but the total computation needed to keep re-checking that growing cache scales quadratically. In plain terms, double the conversation length and you don’t just double the cost, you roughly quadruple it. At the scale of a multi-trillion-parameter model with a million-token context window, that cost becomes something only the largest labs can absorb. To build a genuinely massive model without that cost exploding, Moonshot needed the model to forget selectively, rather than hoarding every detail forever. In October 2025, Moonshot released Kimi Linear , a smaller 48-billion-parameter model built to test this idea. This is where they developed Kimi Delta Attention KDA , the mechanism that would later be scaled up to power K3. Here’s a simple way to picture the difference between standard attention and KDA: Using this approach, Kimi Linear cut memory use by around 75% compared to standard attention, with only a small trade-off in raw capability. The mechanism behind KDA is called the Delta Rule , and it is what lets the model manage a fixed-size memory instead of a constantly growing one. Instead of stacking new entries on top of old ones, the Delta Rule updates memory in place. If a meeting moves from Tuesday to Thursday, the model does not keep both facts around. It erases “Tuesday” and writes “Thursday” into the same memory slot. Two learned values control this process for each attention head: KDA takes this further by giving each attention head 128 separate forgetting dials, rather than one shared setting. This lets the model be very selective. It can drop small talk and filler almost immediately, while holding on tightly to a specific name or date from hundreds of thousands of tokens earlier in the conversation. This “erase then write” approach also prevents new information from interfering with older memories, which was one of the main weaknesses of earlier linear attention designs. KDA itself builds on earlier research into Gated DeltaNet, which first introduced the idea of learning how much information to write into a recurrent memory state. Kimi K3 is a 2.8-trillion-parameter model, currently the largest open-weight model released, and Moonshot describes it as the first open model in the 3-trillion-parameter class. It supports a 1-million-token context window and understands text, images, and video within a single model, using a vision encoder called MoonViT-V2 about 401 million parameters to handle image and video input, alongside a roughly 160,000-token vocabulary. A few architectural pieces work together to make this possible. Kimi Delta Attention KDA , for most layers. According to Moonshot’s technical report, K3 has 93 attention layers in total, and 69 of them use KDA to keep memory use flat regardless of context length. Moonshot reports that this delivers noticeably faster decoding at million-token context lengths compared to standard attention. Gated Multi-Head Latent Attention Gated MLA , for the rest. KDA alone risks losing exact details over very long stretches, since it is built for efficient forgetting rather than perfect recall. So the remaining 24 layers use Gated MLA instead, an evolution of the Multi-Head Latent Attention architecture originally introduced by DeepSeek, which compresses the KV cache into a smaller latent representation. In K3, these layers are interleaved with the KDA layers at roughly a 3-to-1 ratio: the KDA layers handle efficient, fixed-size memory, while the Gated MLA layers act as a compressed full-recall path that preserves exact, token-by-token history for details that cannot be allowed to fade. Attention Residuals AttnRes . In deep models, information from early layers can get diluted as it passes through many later layers, sometimes called PreNorm dilution. AttnRes acts as a drop-in replacement for standard residual connections, letting later layers selectively pull in representations from earlier layers instead of relying only on a single accumulated stream. Moonshot reports this delivers around 25% better training efficiency for well under 2% additional compute cost. Stable LatentMoE. K3 uses a Mixture-of-Experts design with 896 experts, of which only 16 are activated per token. Out of the 2.8 trillion total parameters, Moonshot’s technical report states that roughly 104 billion are active for any given token. This sparsity is what allows the model to reach such a large total parameter count while keeping the actual compute per token manageable. Put together, Moonshot says these architectural changes, combined with updates to training and data, give K3 roughly 2.5x the overall scaling efficiency of K2, meaning it converts a given amount of compute into more usable capability than its predecessor. Running stable training at 2.8 trillion parameters with only 16 of 896 experts active required a few extra tricks beyond the core attention design. Quantile Balancing. With so few experts active per token, keeping the workload evenly spread across all 896 experts is a real challenge. Instead of the usual approach of adding an auxiliary loss term to nudge routing toward balance, Quantile Balancing sets each expert’s routing bias directly from router-score quantiles, so load balancing falls out of the routing math itself rather than needing a separate, sensitive hyperparameter to tune. Per-Head Muon. Muon is a training optimizer that has become popular for large-scale model training. Moonshot extended it so that attention heads are optimized independently rather than as one block, which the team says gives more adaptive learning at this scale. Sigmoid Tanh Unit SiTU . This is a custom activation function used in place of more common choices like GeLU or SwiGLU, intended to give the model finer control over activations. Quantization-aware training. Rather than training at full precision and quantizing afterward, K3 is trained with quantization awareness starting from the supervised fine-tuning stage. The released weights use the MXFP4 format with MXFP8 activations, which keeps the model runnable on a wider range of hardware without a separate, lossy quantization step after training. At launch on July 16, 2026, Kimi K3 took the top spot on Arena’s WebDev leaderboard a human blind-vote benchmark for AI-generated front-end code with a score of 1,679, ahead of Claude Fable 5 and GPT-5.6 Sol at the time. It also led on several sustained coding and agentic benchmarks, including Program Bench, SWE Marathon, BrowseComp, and OmniDocBench. Moonshot has been direct that K3 does not lead across the board. The company’s own materials describe it as trailing the strongest proprietary models, including Claude Fable 5 and GPT-5.6 Sol, on overall performance and on benchmarks like FrontierSWE and HLE-Full, even while outperforming other tested models on its evaluation suite. As is typical with vendor-reported benchmarks, results depend heavily on the exact test harness, reasoning settings, and context management used, so these numbers are best read as a general signal of strength rather than an exact ranking. Kimi K3 comes from Moonshot AI, a Beijing-based company founded in 2023 by Yang Zhilin, who studied computer science at Tsinghua University, completed a machine learning PhD at Carnegie Mellon, and worked on earlier long-context research such as Transformer-XL and XLNet before starting Moonshot. Long-context modeling has been a consistent focus for the company since its earliest products, and KDA is best understood as the latest step in that same line of work. K3 was announced on July 16, 2026, with the full open-weight release following on July 27, 2026, distributed as roughly 96 shards totaling around 1.56 terabytes, under a custom Kimi K3 License. It is available through Kimi.com, Kimi Work, Kimi Code, and the Kimi API, and Moonshot has also contributed a KDA implementation to the vLLM community to support self-hosted deployment. The throughline across K1.5, K2, K2.5, Kimi Linear, and now K3 is a shift away from “just add more compute” and toward teaching models to manage memory the way a person might: keep the important stuff precise, let the unimportant stuff fade, and avoid paying to re-read the entire conversation from scratch every time something new is said. That combination of selective memory KDA , targeted exact recall Gated MLA , better information flow across depth AttnRes , and aggressive but carefully balanced sparsity Stable LatentMoE is what let Moonshot push past the 2-trillion-parameter mark on open weights while keeping the model usable at a million tokens of context. Inside Kimi K3: How Moonshot AI Built the Largest Open-Source Model https://pub.towardsai.net/inside-kimi-k3-how-moonshot-ai-built-the-largest-open-source-model-c4af3555e73a was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.