# Kimi K3: What a 2.8T Open Model Changes for Engineers > Source: > Published: 2026-07-29 00:00:00+00:00 ## On this page # Kimi K3: What a 2.8T Open Model Changes for Engineers I spent a weekend with Moonshot's Kimi K3 launch notes and tech report. The useful story is not leaderboard rank. It is systems: sparse MoE, long context, open weights, and what it takes to run a 3T-class agent brain. I spent last weekend with [Kimi K3](https://www.kimi.com/blog/kimi-k3). Not chatting with it for fun. Reading the launch notes, the [tech report](https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_report.pdf), and the parts people skip once the scoreboard drops. The timelines were loud. *Does it beat Claude Fable 5?* *Does it beat GPT-5.6 Sol?* Those questions matter. They are not the useful ones. K3 is a **2.8-trillion-parameter** mixture-of-experts model with about **104 billion** parameters active per token, native vision, and a **1-million-token** context window. Moonshot calls it the first open 3T-class model. Full weights are out. The API is live. Moonshot itself says overall product feel still trails Fable 5 and Sol. So why bother? Because the interesting claim is **systems**. They rebuilt attention, MoE routing, training, and serving so a sparse 2.8T model can train, run long agent jobs, and ship as open weights. I have been circling the same idea for months: [agent failures are often harness failures](/blog/2026/harness-engineering-reliable-ai-agents/). Reading K3 made that concrete again. This is not just “bigger open weights.” It is an open model that expects a serious environment around it. This post is that weekend read, turned into a decision guide. I start with the specs that matter, then the architecture behind the scale, then how Moonshot trained for hardware instead of leaderboards alone. From there: benchmarks with harness caveats, cost and openness vs Fable 5 and Sol, the long-horizon case studies, what self-hosting really costs, and when I would pick K3. ## The Numbers That Matter Ignore the marketing line for a minute. Start with the specs: | Spec | Kimi K3 | Why it matters | |---|---|---| | Total parameters | 2.78T | Size of the stored model | | Active parameters | ~104B | Cost of each token | | Experts | 16 of 896 routed + 2 shared | Sparse compute, dense memory pressure | | Context | 1,048,576 tokens | Whole repos, long PDFs, multi-hour traces | | Vision | Native, trained with text from the start | Screenshots and video in the same loop | | Scaling claim vs K2 | ~2.5x better efficiency | Not just “more GPUs” | K2 had **1.04T** total and **32.6B** active. K3 jumps to **2.78T** total and **104.2B** active. Layers go from 61 to 93. Training context goes from 128K to 1M. Attention moves from MLA-only to a hybrid of *Kimi Delta Attention* and *Gated MLA*. **Why this matters:** a 2.8T dense model would be a different class of machine. A sparse MoE with 104B active is still huge. But it is a design you can talk about *serving*, not only training. Moonshot positions K3 for long-horizon coding, knowledge work, and agent runs. Think multi-hour repo repair. Compiler work. Research pipelines. Not “answer this chat.” ## The Architecture Is the Product Here is where my read shifted. K3 is not a plain Transformer scaled until the cluster cried. The backbone mixes three ideas: - cheaper long-sequence attention - better depth routing - MoE that stays balanced at extreme sparsity ### Kimi Delta Attention KDA is a linear-style attention path meant for long sequences. Instead of growing a full key-value cache forever, it keeps a **fixed-size recurrent state** and updates it token by token with a forget gate. Each block uses three KDA layers, then one Gated MLA layer. The mix is **3:1**. Softmax-style global attention still shows up. Most of the depth runs on the cheaper path. For engineers, the practical effect is *prefix-cache semantics*. KDA does not cache like classic Transformer attention. Moonshot contributed KDA plus prefill-cache support into the vLLM community for that reason. If your serving stack assumes “KV cache as usual,” K3 will fight you. ### Attention Residuals Normal residual stacks squash earlier layers into one running state. AttnRes does something closer to selective recall across depth. Learned pseudo-queries decide how much to pull from the embedding and from earlier block outputs. In distributed-systems terms, this is **selective replay**. Not “append everything to one log and hope compression holds.” Deep models lose detail. AttnRes is an attempt to keep useful earlier state available without paying full dense attention everywhere. ### Stable LatentMoE and Quantile Balancing K3 activates **16 of 896** routed experts per token. At that sparsity, routing is not a side detail. Bad balance kills throughput. Hot experts stall the cluster. Cold experts waste capacity. Stable LatentMoE treats routing as a first-class problem. Quantile Balancing sets expert load from router-score quantiles instead of fragile hand-tuned balance losses. Moonshot also uses SiTU-GLU activations and Per-Head Muon so training stays stable at this scale. ``` php %%{init: {"layout": "dagre"}}%% flowchart TD Token[Token] --> Router[Router scores 896 experts] Router --> QB[Quantile Balancing] QB --> Pick[Pick 16 experts] Pick --> Shared[2 shared experts always on] Pick --> Routed[16 routed experts] Shared --> Out[MoE output] Routed --> Out ``` **Why this works:** large MoEs often fail in ops before they fail in quality. If expert load skews, GPUs sit idle while one expert burns. Designing balance into training is how you keep a 16-of-896 model from becoming a paper weight. ## Training That Knows About Hardware Older open models often train dense or lightly sparse, then bolt on quantization and serving tricks later. K3’s post-training path is more hardware-aware from the supervised-fine-tuning stage onward. From SFT forward, Moonshot uses quantization-aware training with **MXFP4** weights and **MXFP8** activations. The goal is broad accelerator support and lower memory pressure. You do not want to discover quality loss on deploy day. Expert-parallel training uses static shapes and avoids host sync on the critical path. That is a systems sentence with a product consequence: expert imbalance should not quietly destroy tokens-per-second once you leave the research cluster. Data synthesis is also more structured than “scrape the web and hope.” One training path uses a knowledge graph to pick related concepts, fetch public source material, and synthesize tasks by type. Long context is not RoPE stretched as an afterthought. K3 uses **no explicit positional embedding**. Position shows up through KDA’s recurrent gates and decay. That is how Moonshot claims direct extrapolation to 1M tokens. ## Where the Benchmarks Point Moonshot’s launch table is broad: DeepSWE, ProgramBench, Terminal-Bench 2.1, FrontierSWE, SWE Marathon, agent suites, office suites, vision suites. Independent writeups and the tech-report summary put K3 near the frontier. Often second or third. Sometimes first on a specific coding track. A few scores to keep in your head: | Benchmark | Kimi K3 | Notes | |---|---|---| | DeepSWE | 67.5 | Behind Sol 73.0 and Fable 5 70.0 | | Terminal-Bench 2.1 | 88.3 | Essentially tied with Sol 88.8 and Fable 5 88.0 | | ProgramBench | 77.8 | Leads the reported table | | SWE Marathon | 42.0 | Leads the reported table | | FrontierSWE | 81.2 | Behind Fable 5 86.6, ahead of Sol 71.3 | Read those numbers with the harness disclaimer. Moonshot mixes Kimi Code, Claude Code, Codex, mini-SWE-agent, and vendor-reported scores. Some Fable 5 runs include fallbacks. Some Sol runs hit cyber guards. Same-table numbers are useful for *“is this real?”* They are weak for *“ship everything to K3 tomorrow.”* **The key insight:** K3 looks strongest where work is long, tool-heavy, and messy. Short chat polish is not the claim. Multi-hour coding and agent execution is. ## Capability, Cost, and Openness For engineers choosing a default model, three axes matter more than one Elo number. | Dimension | Kimi K3 | Claude Fable 5 | GPT-5.6 Sol | |---|---|---|---| | Context | 1M native | ~1M with compaction strategies | ~1M long-context stack | | Coding posture | Near frontier; leads some long-horizon suites | Often #1 or #2 in Anthropic-friendly harnesses | Leads some same-harness coding suites | | Agentic tools | Strong, not always best | Very strong | Strong, often fast | | Multimodal | Native vision + video | Strong docs and charts | Strong multimodal + OpenAI tooling | | API price / 1M tokens | $0.30 cache hit, $3 miss, $15 out | Roughly $10 in / $50 out in public comps | Roughly $5 in / $30 out in public comps | | Weights | Open | Closed | Closed | The price gap is not abstract. Moonshot says the official API hits **above 90% cache rate** on coding workloads. If that holds in your repo loops, cache-hit input at `$0.30/MTok` changes the economics of long agent sessions. Openness is the other gap. Fable 5 and Sol stay hosted. K3 is the only model in this comparison you can put on your own supernode, inspect, and wire into a private harness. That is the part that stayed with me after the weekend. Not “K3 is always smarter.” More: *“this is the first open model I would seriously evaluate as the brain of a long-running coding agent.”* ## What “Long-Horizon” Looks Like in Practice Benchmarks are abstract. Moonshot’s case studies are concrete. They show the kind of work K3 is built for. **GPU kernels.** Models got up to 24 hours to profile, rewrite, and benchmark kernels across AttnRes, KDA, and MLA on Hopper and another vendor GPU. K3 stayed competitive with Fable 5 and beat several strong closed models. Late in development, an early K3 already did most of the team’s own kernel work. **MiniTriton.** K3 built a Triton-like compiler from scratch: tile IR over MLIR, passes, PTX codegen. On supported roofline tests it matched or beat Triton and `torch.compile` on some workloads, then kept nanoGPT training stable enough to validate the full stack. **Chip for a nano-model.** In one 48-hour autonomous run with open-source EDA tools on Nangate 45nm, K3 designed and checked a chip for a nano model based on its own architecture. Reported result: **4 mm²**, **100 MHz** timing closure, over **8,700 tokens/s** decode in simulation. **Astrophysics pipeline.** To reproduce I-Love-Q universal relations, K3 reviewed 20+ papers, coded 300+ equations of state, wrote 3,000+ lines of Python, found formula inconsistencies, and shipped an interactive dashboard. Moonshot says that took about **two hours** of agent time versus **one to two weeks** for an experienced researcher. These are vendor stories. Treat them as demos of range, not as a guarantee that your backlog will melt. Still, the pattern is clear: K3 is being sold as an *agent engineer*, not a prettier autocomplete. ## Self-Hosting Is Real, and Expensive Open weights do not mean “runs on my laptop.” Moonshot recommends supernode deployments with **64 or more accelerators** so MoE routing and long-context attention can use large high-bandwidth domains. A practical mental model: ``` Baseline interactive self-host ------------------------------ GPUs: 64x H100 / B200-class (or equivalent) Topology: dense NVLink / NVSwitch-style fabric Runtime: vLLM with KDA kernels + OpenAI-compatible API Layout: expert-parallel shards across nodes Watch: expert skew, KV/prefill cache hits, inter-node bandwidth ``` Software is not plug-and-play either: - You need **KDA-aware** attention and prefix-cache support. - Expert checkpoints must map cleanly onto your topology. - Your agent harness must **preserve thinking history**. K3 was trained that way. Drop the history, or hot-swap mid-session from another model, and quality can go unstable. - Observability must include **expert load**, not only GPU util and tokens/sec. If you do not want that ops bill, use the hosted API. Model ID is `kimi-k3` . Pricing is **$0.30 / $3.00 / $15.00** per million tokens for cache-hit input, cache-miss input, and output. ## Failure Modes You Should Plan For Moonshot is unusually direct about limits. Keep these next to the scoreboard. **Thinking-history sensitivity.** Wrong harness, missing history, or mid-session model swap can make generation unstable. Prefer a verified harness such as Kimi Code.**Too much initiative.** Long-horizon training makes K3 push ahead when intent is vague. If you need narrow behavior, put hard rules in the system prompt or`AGENTS.md` .**UX gap.** Competitive benchmarks do not equal best chat feel. Fable 5 and Sol still win on polish for many users.**Harness effects.** Different agent wrappers change scores. Compare same harness before you rewrite your stack.**Self-host cost.** Open weights remove vendor lock on the model. They do not remove the cluster bill. This is where K3 connected back to my earlier writing. I have argued that [agent failures are often harness failures](/blog/2026/harness-engineering-reliable-ai-agents/). K3 makes that concrete. The model expects preserved thinking state and clear boundaries. If your environment lies to it, the model will look worse than it is. ## When to Choose What **Choose Kimi K3 if you:** - Need open weights at frontier-adjacent quality - Run long coding or research agents where cache hits are high - Care about data residency, custom harnesses, or on-prem serving - Want native vision in the same model that writes code **Choose Claude Fable 5 if you:** - Optimize for product feel and strong Anthropic-harness coding results - Prefer a closed API with less serving complexity - Need top scores on suites where Fable currently leads **Choose GPT-5.6 Sol if you:** - Live in the OpenAI tool stack - Want strong coding and agent speed with closed weights - Prefer OpenAI’s ops and safety surface over self-hosting **Stay on a smaller open model if you:** - Cannot fund a 64-GPU-class deployment or K3 API spend - Mostly need short chat, light coding, or offline demos ## The Bottom Line Kimi K3 does not end the closed-model era. It changes the open-model ceiling. The useful story is not a single win on ProgramBench or SWE Marathon. It is a set of systems choices: hybrid KDA attention for long context, Attention Residuals for depth, Stable LatentMoE with Quantile Balancing, quantization-aware training, and serving work that makes a 2.8T sparse model usable as an agent brain. For engineers, the decision is practical. Use the API when you want near-frontier long-horizon coding without owning MoE ops. Self-host when open weights, control, and private data beat the cluster cost. Keep Fable 5 or Sol when polish and closed-stack convenience matter more than openness. K3 is not “the only model you need.” It is the first open model I would seriously evaluate as the core of a long-running coding agent, with eyes open about harness design and hardware reality. *Evaluating frontier models for coding agents, or trying to decide between open weights and closed APIs? Tell me what constraint bites first: cost, latency, data residency, or harness quality. Reach out on LinkedIn.*