Part 1 was about capacity: how K3 grew to 2.8 trillion parameters without activating all of them for every token. Part 2 was about information flow: how Kimi Delta Attention made million-token memory affordable, why Gated MLA still appears periodically, and how Attention Residuals move information across 93 layers.
At the end of Part 2, we had something that looked like Kimi K3. But it still wasn’t really Kimi K3. We had the architecture.
An architecture tells you how information can move. It tells you which experts can activate, how sequence memory works, how global attention re-enters the stack, and how representations can travel through depth. It does not tell the model what an image means. It does not teach it how to use a million-token context. And it definitely does not teach it how to open a repository, run a tool, see that something failed, fix it, and continue working.
That part comes from training. And K3’s training story is interesting because Moonshot did not treat pretraining, reasoning, agents, and deployment as independent stages bolted onto each other at the end. They were designed to fit the machine they had just built.
The high-level path looks like this: Architecture → Pretraining → Base Model → SFT → Reinforcement Learning → MOPD → Unified Kimi K3. The architecture creates the possibility. Training has to turn that possibility into behaviour.
This distinction sounds obvious, but it becomes important with a system as complicated as K3. Stable LatentMoE, KDA, Gated MLA and Attention Residuals are architectural mechanisms — they determine how computation happens. Pretraining determines what the weights inside those mechanisms actually represent.
Take KDA. Part 2 explained how it gives K3 a fixed-size recurrent state instead of forcing every layer to maintain an ever-growing conventional KV history. That makes extremely long sequences much cheaper. But KDA itself does not teach the model that a variable declared 300,000 tokens ago might matter to an error appearing near token 900,000. The architecture only gives the model a way to carry information that far. Training has to teach it when that information matters.
The same distinction appears everywhere in K3. The architecture gives the model 896 routed experts; training gives those weights useful structure. The architecture gives K3 a million-token context window; training teaches it to actually look across that window. And the architecture gives it enough capacity to run extremely long agent trajectories; post-training teaches it what to do during those trajectories.
Architecture— defines whatcanhappen: routing, memory, depth, context length — defines whatdoeshappen: the weights, the behaviour, when to use it
So the first part of this story is simply building the base.
K3 is natively multimodal, but the interesting part is not that it can accept images — plenty of models can do that. The interesting part is when vision enters training.
A common path to a multimodal LLM is to take an already capable language model, attach a pretrained vision encoder, then perform additional training to make the two systems understand each other. Moonshot experimented with a pretrained vision encoder too. But K3’s final vision encoder, MoonViT-V2, was trained from scratch alongside the language model.
Image → MoonViT-V2 → Visual representations → K3 backbone ← Text tokens
The language model and the vision system therefore learn together rather than meeting only after one side has already finished learning its representation space.
Moonshot reports that the from-scratch vision encoder was actually easier to optimize jointly. The pretrained baseline produced larger and spikier gradient norms during training, while MoonViT-V2 trained more stably and still reached comparable visual performance.
That makes “native multimodality” less of a product label and more of a training decision. Vision is not an adapter hanging off the side of K3 — it participates in building the base model. And the objective itself remains surprisingly ordinary: given everything seen so far, predict what comes next. Sometimes that context is text. Sometimes part of that context came from an image. Both eventually enter the same autoregressive backbone.
Part 2 solved the architecture side of the long-context problem. KDA makes recurrent memory cheap. Gated MLA periodically gives the model global token-to-token interaction again. But none of that means you can change a configuration value to “1,000,000” and suddenly have a model that understands a million tokens.
K3 did not begin pretraining at that length. Moonshot increased context progressively. The model first learns most of its general representations at much shorter sequence lengths. During main pretraining, the context grows from 8K to 64K. The much more expensive extensions come later, first to 256K and eventually to one million tokens.
There is a second problem hiding here. A model can technically train on a million-token sequence without learning much about million-token reasoning. Imagine giving it a huge document where the information needed to answer every question happens to sit in the final few thousand tokens. The sequence is long. The dependency is not.
So Moonshot also creates long-context training examples where useful information is deliberately scattered across distant parts of the sequence. Instead of a document where the answer sits conveniently nearby, the model gets something closer to: Clue A, then a long gap, Clue B, another long gap, Clue C, another long gap, then a question that requires combining A, B, and C to reach the answer.
Now length actually matters. The model cannot solve the task by paying attention only to the local neighborhood around the question. It has to carry information forward, recover distant evidence, and combine pieces that may be separated by hundreds of thousands of tokens.
This closes the loop from Part 2. KDA and Gated MLA make million-token context computationally practical. The long-context curriculum teaches K3 how to use it. Those are two different problems. K3 needed both solved.
Once pretraining finishes, K3 has broad language, visual, and long-context capabilities. But that still does not give you a good assistant or agent. A base model knows patterns. A deployed model has to behave. It has to follow instructions, decide how much reasoning a task needs, use tools correctly, recover when an action fails, work inside software environments, and continue coherently across long trajectories.
Moonshot begins that transition with Supervised Fine-Tuning, or SFT: Pretrained K3 → SFT → Initial Policy. Think of SFT as the cold start for the policy. It gives the pretrained model examples of the kinds of behaviours Moonshot wants.
But K3’s more interesting specialization happens after that, through reinforcement learning. And Moonshot does not train one RL policy to do everything.
There are two different variables Moonshot wants to control during reinforcement learning. The first is the kind of task — general problems behave differently from long-running agents, and coding agents behave differently again. The second is reasoning effort — a straightforward request should not necessarily consume the same inference budget as a hard research task or an extended coding job.
K3 therefore divides RL along both axes. The domains are General Tasks, General Agents, and Coding Agents. Each is trained at Low, High, and Max reasoning effort. Put those together and you get nine specialist policies.
General Tasks— Low · High · MaxGeneral Agents— Low · High · MaxCoding Agents— Low · High · Max↓9 RL policies
There is an easy terminology trap here. These nine policies have nothing to do with K3’s 896 MoE experts. The 896 MoE experts live inside the neural network, and the router selects a small subset of them while a token is moving through the model. The nine RL experts are specialist versions of the whole model, created during post-training. One distinction belongs to architecture. The other belongs to training.
Separating the nine policies lets Moonshot specialize behaviour before trying to squeeze everything back into one final model.
Normal reinforcement learning can have a fairly clean loop: the model answers, the environment evaluates the answer, a reward comes back. Agentic RL is messier.
A coding agent might inspect a repository, search for a symbol, open several files, edit one of them, run a test, watch it fail, inspect the stack trace, change the implementation, run the test again, and keep going. The useful training trajectory might be hundreds of steps long.
Inspect → Search → Edit → Run → Observe → Fix → Run again → Continue
Now the model is not the only thing that needs memory. The environment does too. If the sandbox disappears halfway through the trajectory, the next model action no longer has the world produced by the previous one. The filesystem has changed, processes may have run, tools may have produced state, and the agent might be halfway through an implementation.
So Moonshot built the agentic RL infrastructure around persistent rollout and sandbox state. The model can continue the trajectory without repeatedly reconstructing the world it was acting inside. At million-token scale, that distinction matters: a long context keeps the history of what happened, while a persistent environment keeps the consequences of what happened. An agent needs both.
There is another subtle failure mode. Suppose you train an agent entirely inside one coding harness — it sees the same tool schema, the same command conventions, the same observations, the same wrapper around every task. It can start learning the interface instead of learning the underlying behaviour.
Moonshot tries to separate those things with a Unified White-Box RL Environment capable of instantiating multiple agent harnesses.
K3 Agent Policy → Unified White-Box RL Environment → Harness instance(Kimi Code · Claude Code · Codex · OpenClaw · Hermes)→ reason → inspect → act → observe → continue
The harness changes. The underlying loop should not. That is what cross-harness training is trying to force. A model that only knows where Claude Code expects a tool call is less interesting than a model that understands why it needs the tool call and can express that behaviour through a different interface. This is one of the places where K3’s “agentic” label stops being marketing and becomes an actual training-system problem.
The nine RL policies solve specialization. They create another deployment problem: you obviously do not want to serve nine separate 2.8-trillion-parameter checkpoints and choose one every time somebody sends a prompt. The useful behaviours have to come back together.
Moonshot does that with Multi-Teacher On-Policy Distillation, or MOPD. Each specialist has become good at a different combination of domain and reasoning budget. MOPD uses those specialized policies as teachers and consolidates their behaviour into one student policy.
The important thing is what disappears at the end. There are no nine separate user-facing K3s. The specialization was useful during training. The deployed result is one model — one that can then expose different reasoning-effort settings without requiring Moonshot to deploy an entirely different foundational model for every mode.
At this point, the model could theoretically be finished. But K3 is still 2.8 trillion parameters, and serving it efficiently matters just as much as training it.
This creates another mismatch. Large models are often trained using relatively high numerical precision, then quantized afterward so the weights are cheaper to store and execute. That can change the model: a value that existed precisely during training is now represented approximately during inference. Enough tiny differences can become a real quality drop.
K3 tries to reduce that mismatch with Quantization-Aware Training. Importantly, QAT starts from the SFT stage rather than being applied only after post-training is complete. For K3’s expert computation, the deployed path uses MXFP4 weights with MXFP8 activations.
The idea is easier than the terminology.
Traditional:Train → Finish → Quantize → DeployK3:SFT + RL (trained with deployment precision) → Model adapts → Deploy
K3 moves the deployment constraint into training: SFT and RL happen while training with the expected deployment precision, so the model adapts before it ever ships. The deployment optimization is no longer completely separate from model training — it becomes part of it.
Quantization makes each model pass cheaper. But autoregressive decoding still has an annoying property: tokens come out sequentially. The model predicts one token, then the next, then the next. For a model this large, repeatedly invoking the full network is expensive even if only part of its MoE capacity activates.
Speculative decoding attacks that from a different direction. Instead of asking full K3 to independently produce every single next token, a much cheaper draft model proposes several tokens ahead. If the draft is right, K3 can accept multiple proposed tokens at once. If it is wrong, the full model corrects the sequence.
Draft model → proposes t1 t2 t3 t4 → K3 verifies → accept several → continue
K3 already had a useful component for building that draft model: its multi-token-prediction machinery from pretraining. Moonshot converts that into an EAGLE-3-style speculative decoding draft model and trains it specifically to make proposals the full model is likely to accept. That matters because speculative decoding only helps when the draft model is both cheap and correct often enough. The goal is not to make another smart model — the goal is to build a good guesser for K3.
At this point, the full story becomes much easier to see. Parts 1 and 2 described what happens inside K3 when a token is moving through the model. Part 3 explains where the behaviour inside those weights came from.
K3 Architecture → Native text + vision pretraining → 8K→64K→256K→1M context extension → Base Model → SFT → 9 RL specialist policies → MOPD → Unified K3 → QAT-aware deployment → EAGLE-3 speculative decoding
And this is where the three parts of the series finally connect. Part 1 asked how you build a much larger model without paying for all of it on every token — the answer was sparsity and Stable LatentMoE. Part 2 asked what happens when you stretch that model across a million-token sequence and 93 layers of depth — the answer was KDA, periodic Gated MLA, and Attention Residuals.
Part 3 asks the final question: once all of that architecture exists, how do you teach it to actually become useful?
You train vision with the base model rather than attaching it at the end. You grow context progressively instead of pretending one million tokens is just a configuration switch. You make the training tasks require genuinely distant information. You split reinforcement learning by domain and reasoning effort. You give long-running agents persistent environments. You train across harnesses so the wrapper does not become the skill. You let specialist policies learn separately, then distill them back into one model. And you expose deployment constraints during training instead of discovering them after the model is finished.
That pattern should feel familiar by now. K3 keeps finding the next bottleneck, then it changes the system around it. A bigger model creates a compute bottleneck. A sparser model creates a communication bottleneck. Longer context creates a memory bottleneck. Greater depth creates an information-flow bottleneck. Long-horizon agents create an environment bottleneck. Specialized RL creates a consolidation bottleneck. Deployment creates precision and decoding bottlenecks.
Kimi K3 is not really one architectural trick. It is a chain of engineering decisions where solving one scaling problem reveals the next one. And that is probably the cleanest way to understand the whole model.
Part 1 was about scaling capacity. Part 2 was about scaling memory and depth. Part 3 is about teaching that architecture to see, remember, reason, act — and actually run in production.
Credits:This write-up was developed in collaboration withShivam Misrahttps://medium.com/@shivammishrrr, whose insights and contributions helped shape the analysis and discussion.
From Pretraining to Agentic K3: How Moonshot Trained the Architecture was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.