AIArticle Moonshot's new flagship validates hybrid attention, latent MoE, and 4-bit training at frontier scale — and redefines open weights.
Priya Nair Moonshot AI shipped the full weights for Kimi K3 yesterday, eleven days after the API went live. At 2.8 trillion total parameters it's comfortably the largest open-weight model ever released — nearly triple Kimi K2's 1T — and the headline number is the least interesting thing about it.
The interesting thing is what the parameter count is made of. Read through the architecture, as Sebastian Raschka did in his widely circulated notes, and a pattern emerges: nearly every block of the standard transformer has been swapped for a compressed or linearized variant. K3 isn't a research statement. It's the first frontier-scale model built almost entirely out of the efficiency tricks the field has been accumulating since 2024 — and that's exactly what makes it worth studying.
A transformer where everything got replaced #
Start with attention. K3 runs 93 layers, and only 24 of them use full attention — and even those use gated multi-head latent attention (MLA), the KV-compression scheme DeepSeek introduced with V2. The other 69 layers run Kimi Delta Attention (KDA), the linear-attention variant Moonshot debuted in its 48B Kimi Linear model last fall. That's roughly a 3:1 hybrid ratio, the same recipe Kimi Linear validated at small scale. K3 is Kimi Linear industrialized: the pilot plant became the refinery.
The MoE layer got the same treatment. K3 uses what Moonshot calls Stable LatentMoE — 896 experts with 16 routed plus 2 shared experts active per token, where the expert projections are factored through a 3,584-dimensional latent space, roughly the same down-project-then-up-project move MLA applies to attention, and one Nvidia explored in its Nemotron line. Moonshot claims the combined stack delivers about 2.5× better scaling efficiency than K2's architecture. That's a vendor number from the technical report, but the direction is consistent with what the pieces do individually.
Then there's the choice that will launch a hundred ablation papers: K3 drops positional embeddings entirely. No RoPE anywhere — NoPE across all 93 layers, at 2.8T parameters, with a 1,048,576-token context window. NoPE has floated around the literature since 2023 as a length-generalization curiosity, and hybrid architectures had already been shrinking RoPE's footprint. But as Raschka notes, K3 is the first frontier-level model to commit to it wholesale. If a 1M-context flagship works without positional embeddings, the burden of proof just flipped onto RoPE.
The one genuinely new component is what Moonshot calls Attention Residuals: skip connections that route across layers, weighted by attention scores rather than summed blindly. Per the technical report, it costs about 4% extra training compute and 2% at inference for consistently better validation loss. It's the only part of K3 that isn't a scaled-up version of something already proven — and notably, it's the one place Moonshot spent compute rather than saving it.
What it takes to actually run this #
Here's where the story turns for practitioners. K3 ships as MXFP4 weights with MXFP8 activations, quantization-aware trained from the supervised fine-tuning stage onward — the same playbook as K2 Thinking's INT4 QAT, moved to the microscaling formats that NVIDIA Blackwell and AMD's MI400 series execute natively. Four-bit isn't a community afterthought here; it's the release format, following the precedent OpenAI set with gpt-oss.
Even at 4 bits, the weights are about 1.4TB. A realistic self-hosting floor is an eight-node cluster of 8×80GB GPUs — and that's before you provision KV cache for million-token contexts, which is precisely what the 69 KDA layers exist to keep bounded. For virtually every team, "open weights" here means the API (listed at $3 per million input tokens, $15 output, with cached input at $0.30) or a hosted provider like OpenRouter. The license is MIT-derived and permissive for internal use, modification, and commercial deployment, but reportedly requires a separate agreement with Moonshot once you're reselling inference access past roughly $20M in annual revenue — read it before building a serving business on it.
The economics explain the architecture. At $0.30 cached input and a 1M-token window, Moonshot is pricing for long-horizon agent loops — the workload where you replay an enormous, mostly-static context every step and where linear attention's cheap decode actually compounds. If your workload is agentic coding with a fat repo in context, K3's design is aimed directly at you. If you wanted a model to fine-tune on a node in your rack, this release gives you nothing; that's what the 48B Kimi Linear tier is for.
Open weights are becoming open recipes #
Call it what it is: a genuine shift, but not the one the parameter count suggests. The shift is that the post-2024 efficiency stack — hybrid linear attention, latent factorization of both attention and experts, aggressive QAT, and now NoPE — is no longer a collection of promising papers. It's validated, together, at frontier scale, with weights anyone can inspect. That de-risks the entire recipe for every lab building at 30B or 100B, the scales where most of us actually deploy.
At the same time, K3 marks the point where "open weight" stops meaning "runnable." A 1.4TB artifact that needs a GPU cluster is open the way a blueprint for a container ship is open. The practically valuable release isn't the safetensors files — it's the technical report and the existence proof. Kimi Linear preceded K3 by nine months; the smart money says the K3 recipe shows up in a model you can fit on one node within a year, from Moonshot or from whoever reads the report first.
Raschka called it "a really great release overall," and that's right — but the greatness is in the documentation, not the download. Grab the API for long-context agent work, grab the report for everything else, and let someone with eight nodes worry about the weights.
Sources & further reading #
[Kimi K3 Architecture Overview and Notes](https://sebastianraschka.com/blog/2026/kimi-k3-architecture-notes.html)— sebastianraschka.com -
[MoonshotAI/Kimi-K3: Open Frontier Intelligence](https://github.com/MoonshotAI/Kimi-K3)— github.com -
[Kimi K3 Model Overview: MXFP4 Quantization and Open Weights](https://huggingface.co/blog/ResterChed/kimi-k3-model-overview-mxfp4-quantization-open-wei)— huggingface.co -
[Kimi K3 Open Weights Shipped: What the Licence Says](https://www.digitalapplied.com/blog/kimi-k3-open-weights-shipped-license-restrictions-2026)— digitalapplied.com -
[Kimi K3 - API Pricing and Benchmarks](https://openrouter.ai/moonshotai/kimi-k3)— openrouter.ai
[Priya Nair](https://sourcefeed.dev/u/priya_nair)· AI & Developer Experience Writer
Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.
Discussion 0 #
No comments yet
Be the first to weigh in.