# The 2.8-Trillion-Parameter Problem: Why Bigger Models Break the Old Playbook > Source: > Published: 2026-08-21 03:16:16+00:00 Part 1 of Inside Kimi K3 — a series on how a 2.8T-parameter open model was engineered to actually be servable Here’s a question that sounds simple and isn’t: how do you scale a language model to 2.8 trillion parameters without making every single token it generates prohibitively expensive? That’s not a benchmark question. It’s an engineering question, and it’s the one this whole series is built around. Kimi K3, released by the Kimi Team, is a decoder-style foundation model — the same basic family as GPT, Claude, and Llama — but stretched to a size that, until recently, only existed inside closed labs. To get there without producing something unshippable, almost nothing about the standard Transformer decoder was left untouched: not the feed-forward layer, not the attention mechanism, not even the residual stream that carries information from one layer to the next. This series maps each of those redesigns to the specific problem it solves. No mechanism shows up until you’ve seen why the plain version breaks first. By the end, you’ll have a working mental model of how a token actually moves through K3 — and, more usefully, *why* it moves the way it does. One thing this series won’t do much of: lead with leaderboard rankings. Benchmarks are interesting, but they don’t explain anything. The architecture does. Before touching anything K3-specific, it’s worth being precise about the baseline, because every later section is a deviation from it. A decoder-only Transformer’s forward pass is almost embarrassingly simple to describe at a high level: ``` tokens → embeddings → [decoder block] × N → output head → next-token probabilities ``` Inside a single block, two things happen, each wrapped in a residual connection and a normalization step: Stack enough of these blocks, and you get a model that can write code, hold a conversation, or plan a multi-step task. It’s a good design, which is exactly why K3 keeps its skeleton. What changes is *how* attention is computed, *how* the feed-forward step is computed, and *how* information is allowed to flow between layers. Keep that skeleton in your head — attention, residual, FFN, residual — because this whole series is going to modify one limb of it at a time. A dense decoder scales in a straightforward, brutal way: double the parameters, and you roughly double the compute needed for every single token, because every parameter gets touched on every forward pass. That arithmetic is fine at a few billion parameters. At 2.8 trillion, it stops being fine — you simply cannot afford to activate the entire network for every token you generate. That’s the first problem: **too much compute to activate**. The second problem shows up the moment you try to fix the first one by making the model sparse — using only a slice of the parameters per token — because a model that size doesn’t fit on one accelerator, or even a handful of them. The weights are spread across a rack of GPUs, which means whichever piece of the network a given token needs to use is very likely sitting on a *different chip* than the token itself. Every token now has to travel across an interconnect, get processed, and travel back. That’s **too much communication between accelerators**, and it’s a cost that has nothing to do with arithmetic — it’s a data-movement problem, and at large scale it can dominate the compute problem it was supposed to fix. The third problem is about context length rather than parameter count. Standard self-attention lets any token look back at any previous token with full precision, but the cost of doing that grows fast — double the context, and the attention computation roughly quadruples, while the memory needed to store every past key and value (the “KV cache”) grows right along with it. That’s manageable for a short chat. It is not manageable for a coding agent chewing through a large repository, a long tool-call history, and a sprawling reasoning trace — exactly the kind of workload K3 is built for, with a context window trained out to one million tokens. **Full attention at that length is not a product; it’s a line item nobody wants to pay.** The fourth problem is quieter, and it’s about depth rather than width. K3 is 93 layers deep. In a standard residual stream, each layer reads what the layers before it produced, adds its own contribution, and hands the combined result to the next layer — a relay race. By the time information reaches layer 90, it has been rewritten, blended, and diluted through 89 intermediate handoffs. Useful signal from early in the stack can simply get lost in the shuffle. That’s **information degrading across depth**, and it’s a problem that gets worse, not better, the deeper you go. Four problems, one underlying tension: raw scale versus the practical cost of using that scale. Here’s the map the rest of this series follows to resolve it: Scaling problem K3’s answer Covered in Too many parameters to activate per token Sparse Mixture of Experts Rest of this part Too much expert traffic between accelerators Stable Latent MoE Rest of this part Full attention too expensive at long context Kimi Delta Attention (KDA) Part 2 Linear attention loses exact retrieval Hybrid KDA + global attention Part 2 Information degrades across depth Attention Residuals Part 2 Each of these is, in isolation, a reasonably well-known idea in the current research landscape. What makes K3 worth studying closely is that all four are stacked together, deliberately, inside a single 2.8-trillion-parameter model, and the paper is unusually specific about *why* each one was needed rather than just *that* it was used. The rest of this part tackles the first two problems — too much to activate, too much to ship between chips — and the mechanism K3 built to solve both at once: a sparse Mixture of Experts with a compression trick bolted on. *From a dense feed-forward layer to 896 experts, and the communication bill that sparsity doesn’t tell you about.* Before getting into mechanisms, it’s worth just looking at the numbers, because they tell you where Moonshot chose to spend the extra scale. The single most telling number in that list is the hidden dimension: **7,168, unchanged from K3’s predecessor.** The model didn’t get *wider*. It got *deeper* (61 layers became 93) and its expert pool got *bigger* (384 experts became 896), while the fraction of the model actually switched on for any one token went slightly *down*. That’s not an accident — it’s the whole design philosophy in one row of a spec table: total capacity and per-token compute are being deliberately pulled apart. Recall the plain decoder block from above: attention, then a dense feed-forward network applied to every token. Mixture of Experts (MoE) replaces that single dense FFN with a *bank* of smaller FFNs — “experts” — plus a lightweight router that looks at each token and decides which experts should handle it. Crucially, nobody hand-assigns what each expert specializes in. Specialization is *learned* — the router and the experts co-adapt during training, and useful divisions of labor emerge on their own. What the architecture guarantees is the mechanism, not the meaning: 896 experts exist, and for any given token, only 16 of them get touched. That’s roughly 1.8% of the routed-expert pool doing work at any instant — a small slice of an enormous pool, which is exactly how a model can have trillions of stored parameters while keeping the compute for each token bounded. Here’s the part that’s easy to miss if you only think about MoE in terms of FLOPs: sparsity solves the *arithmetic* problem, but it can quietly create a *communication* problem that’s just as expensive. A 2.8-trillion-parameter model does not fit on one accelerator — not even close. Its weights, including all 896 experts, are physically spread across many GPUs. So when the router picks 16 experts for a given token, there’s no guarantee any of them live on the same chip the token started on. In the general case, the token’s full hidden-state vector has to travel out over the interconnect to wherever each selected expert lives, get processed, and travel back. Think of it like a warehouse system: sparsity shrinks the work happening *inside* any one warehouse, but it can just as easily inflate the shipping bill *between* warehouses. Past a certain scale, a very sparse model stops being bottlenecked by compute and starts being bottlenecked by how much data is crossing the network — and at 896 experts, that pressure is real. K3’s answer is what the paper calls **Stable LatentMoE**, and the core move is exactly what the warehouse analogy suggests: compress the shipment before sending it, and only expand it back once it’s home. Instead of sending a token’s full 7,168-dimensional hidden state to a remote expert, K3 first projects it down into a much smaller **3,584-dimensional latent space** — exactly half the width. The selected routed experts do their work in that compact latent space. Their outputs are combined, then projected back up to the full 7,168 dimensions before rejoining the rest of the network. Worth being precise about one thing here: **this doesn’t mean the whole model runs at 3,584 dimensions.** The main residual stream — the backbone every layer reads from and writes to — stays at 7,168 dimensions throughout. Only the *routed-expert path* gets compressed, and only for the duration of the expert computation. That compression cuts two costs at once: less data has to cross the network to reach an expert, and each expert does its arithmetic on a smaller vector once it gets there. It’s not a free lunch, though. Squeezing an enormous, sparsely-activated routed path through a narrow latent bottleneck turns out to destabilize training in its own way — the paper describes exploding internal activations in the routed branch at this scale. K3’s fix is a small stack of stabilizers: an RMSNorm inserted right before the up-projection back to full width, and a custom activation function the team calls **SiTU-GLU**, which smoothly caps how large the gate and value branches of the expert FFN are allowed to grow (bounded to roughly ±100 in magnitude) while still behaving almost identically to the popular SwiGLU activation near zero. Reliability engineering, in other words, all in service of making the compression trick actually trainable at 2.8T scale. There’s a second failure mode lurking in any large MoE, and it has nothing to do with communication: **the router can develop favorites.** If a handful of experts get routed to constantly while the other 800-plus sit idle, you’ve effectively paid the training cost of 896 experts to get the practical benefit of maybe 50. The standard industry fix nudges an expert’s routing score up or down a little at each step, depending on whether it’s over- or under-loaded — a manual, hand-tuned feedback loop that gets fragile fast as the expert count grows. K3 replaces the nudging with something more direct: **Quantile Balancing.** Rather than guessing at a step size, it computes routing thresholds straight from the distribution of router scores at each step — reading off, essentially, “what score would an expert need to stay within its fair share of tokens” — no auxiliary loss term, no step-size hyperparameter to tune at scale. That solves balance at the *routing* level. There’s a complementary fix at the *infrastructure* level, worth distinguishing clearly: a system called **MoonEP**, which dynamically replicates overloaded experts across accelerators so that even if the router does end up favoring certain experts in a given batch, no single GPU gets stuck doing disproportionately more work than its neighbors. Routing balance and hardware load balance are related problems, but they’re solved by two different mechanisms working together. Put together, sparse MoE plus latent compression plus quantile balancing let K3 store an enormous amount of learned capacity — 2.8 trillion parameters’ worth — while keeping the *active* compute per token down around 104 billion parameters, and keeping the *communication* cost of reaching those parameters compressed by half. That’s the first two of the four scaling problems solved: too many parameters to activate, and too much traffic moving between accelerators. It doesn’t solve the other two. Nothing about MoE touches how expensive attention gets at a million tokens of context, and nothing about it touches the depth problem in a 93-layer stack. Those are next. Credits:This write-up was developed in collaboration withShivam Misra[https://medium.com/@shivammishrrr], whose insights and contributions helped shape the analysis and discussion. [The 2.8-Trillion-Parameter Problem: Why Bigger Models Break the Old Playbook](https://pub.towardsai.net/the-2-8-trillion-parameter-problem-why-bigger-models-break-the-old-playbook-d7619d9c4fb1) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.