An implementation of the architecture, pre-training protocol, systems co-designs, and post-training framework described in Kimi K3: Open Frontier Intelligence (arXiv:2607.24653v1).
Scale is configuration. Nothing in kimi3/
branches on model size, so the same
code path runs a 20M-parameter model on a laptop CPU and describes the paper's
2.8T configuration. This repository executes the small end and the verification
suite. Training at cluster scale is out of scope, and docs/PAPER_MAP.md
marks which components are wired into a runnable path and which are libraries.
Parameter counts reproduce the paper's Table 1:
| This implementation | Paper (Table 1) | |
|---|---|---|
| Total parameters | 2.778 T | 2.78 T |
| Activated per token | 104.108 B | 104.2 B |
| Vision tower | 400.9 M | 401 M |
Python ≥ 3.10, PyTorch ≥ 2.4. No GPU is required. The test suite, the parameter audit of the 2.8T config, the demos, and the nano training run all complete on CPU.
A GPU is used when one is available: resolve_device("auto")
selects CUDA if the card has enough free memory, and falls back to CPU otherwise. Training is roughly 7× faster per step on a small GPU. Triton is optional; the fused KDA path falls back to the reference implementation when it is absent, and never runs during training because it has no fused backward pass.
pip install -e . # runtime
pip install -e '.[dev]' # adds pytest
Optional extras: [hf]
, [vision]
, [triton]
.
python3 scripts/count_params.py configs/k3_2p8t.yaml --check
bash scripts/train_nano.sh
python3 -m kimi3.train.pretrain --config configs/k3_nano.yaml --jsonl corpus.jsonl
bash scripts/verify.sh
Five runnable demos are in demos/
, including a trained 19.8M model generating
Python and a side-by-side comparison of models trained on real versus synthetic
data. See demos/README.md
.
The nano model trains from a loss of 8.13 to 0.32 in 400 steps on synthetic
data, starting at ln(vocab)
as expected for correct initialisation.
On a real corpus -- 47,902 documents of system documentation and source code, deduplicated and split so no held-out document shares a near-duplicate cluster with training data -- 3,000 steps give:
| Metric | Start | End | Reference |
|---|---|---|---|
| Held-out loss | 6.2726 | 0.8434 | 6.238 = chance |
| Parser-valid generations | 0.1389 | 0.7021 | 0.8182 = training corpus |
The model reaches 86% of the parser-validity rate of its own training data, and
remains significantly below it (z = −2.23 over 480 generations). docs/RESULTS.md
Phase 9 documents the scorers, confidence intervals, and why a dictionary word-rate metric is not a quality measure on its own.
Implemented from paper §2:
Kimi Delta Attention with the lower-bounded decay of Eq. 5 and a full-rank output gate.Gated MLA, with NoPE (no positional encoding) throughout.** Block Attention Residuals**.** Stable LatentMoEwith SiTU-GLU and Quantile Balancing. MoonViT-V2**, trained from scratch under next-token prediction.
At the 2.8T configuration:
Hybrid attention schedule. hybrid_ratio: 3
with trailing_global: true
lays out 93 layers as 23 × [KDA, KDA, KDA, MLA] plus a trailing MLA: 69 KDA and 24 MLA layers, so the tail carries two consecutive MLA layers. Under NoPE, position information enters the model only through KDA's recurrent decay.
LatentMoE. 896 routed experts at top-16 (sparsity 56), plus two always-on shared experts and one leading dense FFN layer. Routing is dropless: there is no capacity factor in the config, every token emits exactly top-k assignments, and all of them are computed. Empty experts are skipped; tokens are not. The router scores the full-width d = 7168 token. The 3584-wide latent reduces expert compute and dispatch payload without shrinking the routing space. The Quantile-Balancing bias is a buffer rather than a parameter: mean-centred, applied one optimizer step late, and frozen at inference.
Block Attention Residuals. 8 blocks of 12 layers, so at most eight completed
block sources, one partial sum, and the embedding are resident simultaneously —
O(N·d)
live memory instead of O(L·d)
. Across pipeline stages, AttnResCache
transfers only the blocks the current stage created.
Per-Head Muon, cosine schedule with 1% warmup, the four-stage 8K → 64K → 256K → 1M context curriculum, and document-isolated packing.
KDA Context Parallelism (Eq. 17), MoonEP with the appendix E bound on redundant experts, Pipeline ZeRO-2 with CPU-resident gradient shards, P2P Muon orthogonalisation, block-wise FP8 activations, a unified activation manager, and cache-based AttnRes pipeline communication.
These are verified components, not an assembled distributed trainer. Each is
tested against a single-process reference, and KCP and P2P Muon run under real
torchrun
processes. However, kimi3/train/pretrain.py
is single-process and
imports nothing from kimi3/parallel/
.
Two components are accounting rather than transport:
- MoonEP's dispatch and combine are a local permutation with no
all_to_all
. - The pipeline schedule computes stage partitions and transfer byte counts without moving activations.
docs/PAPER_MAP.md
records the status of each component, and
tests/test_reachability.py
enforces it: a module that nothing calls fails the suite unless it is listed with a reason.
The XTML chat template from appendix F, SFT with MXFP4/MXFP8 quantisation-aware training, partial rollout, per-problem reasoning-effort budgets producing nine domain × effort experts, an agentic generative reward model, Multi-Teacher On-Policy Distillation, and resumable sandboxes.
A unified paged cache for the hybrid KDA–MLA state, KDA-aware prefix caching with
decoupled hash and physical granularities, ReplaySSM speculative decoding, and an
EAGLE-3 draft model trained with the LK loss. A single PagePool
serves both
MLA-KV and KDA-state pages. It is sized by constructor argument, has no YAML
binding and no eviction policy, and raises MemoryError("page pool exhausted")
at the limit.
The reference implementations define the intended behaviour; the kernels are optimisations of them. The main gates are therefore equivalence checks:
| Gate | Result |
|---|---|
| KDA chunkwise form vs Eq. 1 recurrence (fp64) | 6.7e-16, all chunk sizes, with and without document resets |
| KDA Context Parallelism vs single-device states | 4.86e-16 across four processes |
| MoonEP balance within the appendix E bound | 200/200 randomised skewed routings, including all-to-one |
| Block AttnRes at block size 1 vs unblocked Eq. 8–9 | exact |
| Triton kernel vs reference | 5e-4, cosine similarity 1.000000 |
For comparison, the naive additive scheme used for vanilla linear attention is wrong by over 10% on the KCP test case. That error is the reason KCP exists.
Run everything with bash scripts/verify.sh
. This includes 322 unit and
equivalence tests, three multi-process torchrun
gates, and the Triton kernel check.
| Config | Total | Activated | Purpose |
|---|---|---|---|
k3_nano |
|||
| 19.754 M | 8.744 M | correctness proof | |
k3_micro |
|||
| 999.755 M | 273.092 M | multi-process parity, 64K context | |
k3_small |
|||
| 14.452 B | 1.995 B | first multi-node run | |
k3_2p8t |
|||
| 2.778 T | 104.108 B | the paper's configuration |
Every figure is scripts/count_params.py
output rather than an estimate.
docs/SCALING.md
covers choosing PP × EP × CP × DP
degrees and the failure
modes to expect. docs/HARDWARE.md
derives memory, cluster size, interconnect, and cost for each rung.
Training a 2.8T model, or reproducing any benchmark score. The proprietary corpora and in-house evaluation suites are not available.A multi-node launcher. No Slurm, Ray, ortorchrun --nnodes
path exists. The only distributed entry points are the three single-node parity gates inscripts/verify.sh
, which run on gloo.An SFT or RL entry point.kimi3/posttrain/sft.py
andkimi3/posttrain/rl/
export pure functions — losses, schedulers, reward models — with nomain
, no CLI, and no optimizer step. MXFP4/MXFP8 QAT is wired into the expert forward and tested bytests/test_qat.py
, but nothing enables it automatically.MoE all-to-all. MoonEP plans and balances the expert assignment; the token exchange itself is a local permutation.A fused backward pass for the Triton KDA kernel. The kernel therefore never runs during training: the module upcasts to fp32 and the fused path declines any call requiring gradients.The Firecracker backend for AgentENV. What exists is an in-process lifecycle state machine (start//resume/fork/snapshot/destroy
over a dictionary) that executes no code. The repository contains nosubprocess
,exec
, oros.system
call, and theSandbox
protocol has no execute method. A real backend would need to define its isolation contract — process, filesystem, network, timeout, teardown — before running model-generated code.A vision pathway in the trained model.kimi3/vision/
implements MoonViT-V2 and is tested against the paper's parameter count, butkimi3/model.py
does not import it. The backbone is text-only.
| File | Contents |
|---|---|
docs/PAPER_MAP.md |
|
| every paper section mapped to its implementation and test | |
docs/RESULTS.md |
|
| measured results for every gate | |
docs/AMBIGUITIES.md |
|
| the nine fields Table 1 leaves unspecified, and the evidence for each default | |
docs/SCALING.md |
|
| parallelism degrees and failure modes | |
docs/HARDWARE.md |
|
| memory, cluster size, and cost per rung | |
demos/README.md |
|
| five runnable demonstrations |
Copyright © 2026 Tim Rots.
Licensed under the GNU Affero General Public License, version 3 or later ( LICENSE). You may use, study, modify and redistribute this code. If you modify it and make it available to others — including over a network, as a hosted service — you must release your complete source under the same licence.
That is deliberate. The AGPL does not forbid commercial use; it forbids building a closed product on this work. Anyone is free to run it, learn from it, and contribute back.
Commercial licences are available. If the AGPL's source-disclosure requirement does not suit your organisation, contact the author to arrange alternative terms.
The Kimi K3 architecture. This is an independent implementation of a public paper (arXiv:2607.24653v1). The licence covers this source code, not the ideas or the paper, which belong to their authors.Model weights. Any checkpoint produced by training this code inherits obligations from the data it was trained on. The demo weights come from a corpus that is roughly 69 % copyleft-licensed by document count; the legal status of weights derived from such data is unsettled and is not resolved by this licence. No trained weights are distributed in this repository.