Serving Kimi K3 End to End: 8 GPUs, 32 GPUs, and a Rack Serving Moonshot AI's Kimi K3 requires at least 8 B300 GPUs for a single copy because the 2.8-trillion-parameter checkpoint occupies 1.56 TB, against 288 GB per B300 GPU and 141 GB per H200, according to an SGLang-based end-to-end deployment analysis. The analysis lays out three configurations — 8 B300 GPUs (the SGLang cookbook's verified setup, which the author re-measured with real agent requests), 32 H200 GPUs across four servers (run by the author, with configuration and memory arithmetic rather than measurements), and a 72-GPU GB300 rack (a proposal combining pieces measured on smaller systems, with no published full-rack K3 run found). The layout is driven by three numbers: 69 of K3's 93 layers keep no KV cache and hold only one fixed-size state per request, and each MoE module has 896 experts of which a token uses 16, with those experts stored in 4-bit format making up 93% of the checkpoint while all other weights total 107 GiB. There is an interview question I like a lot: “Walk me through serving a production model such as Kimi K3 end to end, once at production size and once with 8 GPUs.” Most answers I hear start from a familiar recipe for large Mixture-of-Experts MoE models. We use tensor parallelism if we care about latency, we switch to DP attention with expert parallelism if we care about throughput, and at scale we split prefill and decoding into separate servers. I described the first two for GLM-5.3 in an earlier article https://how-gpus-talk.vercel.app/ , and for that model, the recipe works. For Kimi K3 https://huggingface.co/moonshotai/Kimi-K3 , the recipe needs more care, because three of K3’s numbers get in the way. First, its checkpoint takes up 1.56 TB 2.8 trillion parameters, by the SGLang team’s count , against 288 GB on a B300 GPU and 141 GB on an H200. That decides how many GPUs one copy needs. Second, 69 of its 93 layers keep no KV cache at all, only one fixed-size state per request, which decides how many requests can run at once. Third, each MoE module has 896 experts, of which a token uses 16. These experts are 93% of the checkpoint, so how we cut them across the GPUs decides every layout below. What does a production deployment of this model look like, then? I want to answer for three sizes: one server with 8 B300 GPUs, 32 H200 GPUs in four servers, and a GB300 rack with 72 GPUs. The same few numbers decide the layout at every size, but something different runs out first each time. I assume you know the mechanisms, because other articles in this series cover them: tensor, data, and expert parallelism, Multi-Head Latent Attention MLA , decode context parallelism, speculative decoding, and disaggregated serving. I don’t have the same evidence for each size: - The deployment on 8 B300 GPUs is the one the SGLang cookbook https://github.com/sgl-project/sglang/blob/f1065423d31efd7df14ef36f58bee00de2c21eed/docs/cookbook/autoregressive/Moonshotai/Kimi-K3.mdx marks as verified, and I measured it again on our machine, with real agent requests. - The deployment on 32 H200 GPUs is one I run myself. I give its configuration and memory arithmetic, not measurements, and I compare it with what others have posted. - The rack deployment combines a proposal for the whole rack with pieces measured on smaller systems. As far as I could find, nobody has published a K3 run that fills a GB300 rack, and I haven’t run one. I stick to SGLang https://github.com/sgl-project/sglang at one commit, NVIDIA GPUs, and text requests, and leave out kernels, cost, and power. 1. What Has to Be Served 1-what-has-to-be-served Before we choose a layout, we need three sizes the weights, the state of a request, and the cache entry of a token and one latency target. 1.1 Where the Bytes Are 11-where-the-bytes-are The model configuration says that K3 has 93 layers with a hidden size of 7,168 and that all layers except the first contain an MoE module. I read the headers of the checkpoint files, which list every tensor with its shape and data type. Figure 1 shows the result. In Figure 1, the routed experts, stored in a 4-bit format, make up 93% of the checkpoint. Everything else is in 16 bits and adds up to only 107 GiB. More than half of that, 57 GiB, is the attention weights of the 69 fixed-state layers, which use Kimi Delta Attention KDA . The 24 layers that use MLA take another 10 GiB. MiB and GiB are powers of 1,024, GB and TB the vendors’ powers of 1,000, so nvidia-smi shows a B300 at 268.6 GiB, about 288 GB. Three things follow from this picture. First, almost all the memory goes to the experts, but we can cut them any way we like. Each GPU can get a slice of every expert, a subset of whole experts, or all the experts of some layers. One layer’s experts take up 14.6 GiB, and a token reads only 16 of the 896. Second, the attention weights are small in comparison, but they get copied if every GPU runs the complete attention module for its own requests, as DP attention does. At 67 GiB 57 + 10 , that copy is expensive. Third, the latent projections map between the model’s hidden size and the smaller size the experts work in. Their 9 GiB are stored in full on every GPU that holds their layer. 1.2 Two Kinds of State 12-two-kinds-of-state Next, let’s look at what the server has to remember for a request. An MLA layer keeps a KV cache, as usual. For every token, it stores one latent vector with 576 values, which all attention heads read. With 24 layers and 16-bit values, a token’s cache entry costs 24 × 1,152 = 27,648 bytes. Since this vector can’t be divided by head, tensor parallelism stores it on each of its GPUs. A KDA layer works differently. KDA is a form of linear attention, where a layer doesn’t compare a new token with all earlier ones. Instead, it folds every token into a matrix of fixed size, one 128 × 128 matrix per head, and reads the new token’s output from it. That’s why a KDA layer keeps one state per request and nothing per token. How large is it? K3 has 96 heads and 69 KDA layers, and the matrices are stored as 32-bit floats by default, which gives 414 MiB. Together with a small buffer per layer for a convolution over the last few tokens, the state of one request takes up 429 MiB. Tensor parallelism does divide it, unlike the MLA cache, since each GPU needs only the heads it computes. On 8 GPUs, that leaves 53.6 MiB per GPU. I will call these 53.6 MiB a state block. One complication is prefix caching, where the server reuses the beginning of a prompt that it has seen before. SGLang can’t share a state the way it shares cache entries, because a state is overwritten with every token. Instead, it keeps copies of the state at fixed positions in the prompt, from which a later request can continue. SGLang’s code budgets 5 state blocks per running request of any length, the live state and these copies 268 MiB per GPU . Let’s take two requests we will use throughout. The fresh request is one coding-agent call, rounded to 8,500 tokens, and the long session has a round 128,000. The fresh request needs 224 MiB of cache per GPU, next to 268 MiB of state. The long session needs 3.3 GiB of cache, next to the same 268 MiB. Figure 2 shows both. The thing to notice in Figure 2 is the crossing point at about 10,160 tokens. Below it, a request costs more state than cache, and above it, the cache dominates. A server full of short requests runs out of state blocks first, and a server full of long sessions runs out of cache first. This crossing point is what K3’s main sizing flag is about. 1.3 Two Pools and One Flag 13-two-pools-and-one-flag SGLang splits whatever is left after the weights into two pools at startup. The state pool holds state blocks. At 5 blocks per request, it limits how many requests can run at once. The cache pool holds the MLA cache entries and is handed out in pages as requests grow. A single flag, --mamba-full-memory-ratio , sets the ratio between the two pools. The cookbook has a calculator for it, and the formula is the comparison from Figure 2. The ratio is the state that an average request needs, divided by the cache it needs. ratio = S + D × state bytes / L × mla kv bytes / DCP + draft kv bytes Here, S is the number of state blocks per request 5 by default , state bytes is the size of one block, and L is the average length of a request in tokens. The remaining terms belong to features from section 2. D is the number of extra state blocks that speculative decoding holds, the draft term is the cache of the draft model, and DCP is the number of GPUs across which decode context parallelism divides each entry’s cost. For example, requests of 8,192 prompt and 1,024 output tokens, 9,216 in total, give a ratio of 1.1 on our 8 GPUs. The SGLang team calls this a bet on the traffic. If requests run longer or shorter than L, one pool fills up while the other has room. 1.4 One Request and a Target 14-one-request-and-a-target Finally, let’s follow one request through a server. The request is rendered into K3’s chat format and tokenized. The scheduler then has to admit it, which needs free state blocks and free cache pages. After that, the prompt goes through all 93 layers in chunks of up to 16,384 tokens prefill , and the server generates the output one step at a time decoding . For the measurements in section 2, I use real agent traffic from the Exgentic v2 traces https://huggingface.co/datasets/Exgentic/agent-llm-traces-v2 , which record every model call in agent sessions, including the system prompt, the tool definitions, and the full history. From them, I built two kinds of requests: - The fresh request: about 8,400 prompt tokens on average with K3’s tokenizer, answered with up to 512 tokens. I tagged each so that it shares no prefix with any other. - The long session: a history of about 118,000 tokens from the same traces, followed by returning turns that each add about 2,000 tokens. As a target, I picked what seems reasonable for an interactive agent. The first token should arrive within 2 seconds. Each stream should then run at 40 tokens per second or more, which means at most 25 ms per token on average over the request. In our runs, about a third of the answers are tool calls that the server delivers in one piece once they are complete. Such an answer meets the target if it arrives within 2 seconds plus 25 ms per token. To test against it, I let requests arrive at random at a fixed average rate. The sustained rate is the highest rate at which 95% of the requests still meet the target. 1.5 Summary 15-summary To summarize, K3’s weights take up 1,454 GiB, of which 93% are 4-bit experts we can cut any way we like. A running request needs 268 MiB of state per GPU and 27,648 bytes of cache per token. One flag splits the server’s free memory into a pool for each, and the state pool limits how many requests can run. 2. Eight B300 GPUs: One Server Does Everything 2-eight-b300-gpus-one-server-does-everything The smallest deployment is a single server with 8 B300 GPUs, for which the cookbook has two recipes it marks as verified by a speed test, not for accuracy . I ran both on our machine with the agent requests from section 1.4. 2.1 The Layout That Fits 21-the-layout-that-fits With 1,454 GiB of weights and 268.6 GiB per GPU, there is room for exactly one copy of the model. The question is how to cut it. For each candidate, I computed the weights per GPU from section 1.1’s tensor list and the way SGLang divides each kind of tensor. Figure 3 shows the result. In Figure 3, tensor parallelism over all 8 GPUs gives every GPU an eighth of the experts and of the attention weights, plus the parts stored in full on every GPU. By my arithmetic, that comes to 192 GiB. The server reported 195 GiB after loading, so the arithmetic is off by less than 2%. I will rely on it for the larger deployments, which don’t fit on this machine. The alternatives don’t leave room to serve. With only 4 GPUs per copy, each GPU would need 372 GiB. With DP attention, the 67 GiB of attention weights are copied to every GPU, and the weights grow to 254 GiB, which leaves almost nothing for state and cache. The recipe is short: sglang serve --model-path moonshotai/Kimi-K3 --trust-remote-code \ --tp-size 8 --mem-fraction-static 0.85 \ --reasoning-parser kimi k3 --tool-call-parser kimi k3 \ --mamba-full-memory-ratio 1.1 SGLang chooses the rest for this model on the B300. It serves the experts in their stored 4-bit format with 8-bit activations, gives each GPU a slice of every expert, and runs the all-reduce sums that tensor parallelism needs through fused kernels the topic of the earlier article https://how-gpus-talk.vercel.app/ . Loading the 1.56 TB took 59 minutes on our machine the first time, and the server answered its first request after 64 minutes. With the checkpoint in the host’s page cache, later starts took 5 to 20 minutes. Either way, we can’t start a server only when the load arrives. 2.2 Sizing the Pools 22-sizing-the-pools --mem-fraction-static 0.85 reserves 15% of each GPU outside the two pools, for activations, buffers, and the kernels’ workspaces. After startup, nvidia-smi showed 232 GiB per GPU: 195 of weights, about 30 of pools, and a remainder the log doesn’t itemize. The ratio of 1.1 divides the pools into 303 state blocks and room for 561,728 tokens. With 5 blocks per request, 303 blocks admit 60 requests at a time. The server’s log says so, with mamba for the state and slot for a block: max running requests is capped to 60 by the mamba state cache max mamba cache size=303, 5 state slots per request . This line is easy to miss, but it states the server’s real concurrency limit, however high we set --max-running-requests . The cache pool holds 66 of our fresh requests, but only 4 sessions of 128,000 tokens. For long sessions, this split is wrong, and so is the way the server stores the cache. The second recipe changes that. 2.3 Low-Latency or Balanced 23-low-latency-or-balanced The cookbook calls the recipe above Low-Latency. The other one, Balanced, adds a single flag and changes the ratio: sglang serve --model-path moonshotai/Kimi-K3 --trust-remote-code \ --tp-size 8 --dcp-size 8 --mem-fraction-static 0.85 \ --reasoning-parser kimi k3 --tool-call-parser kimi k3 \ --mamba-full-memory-ratio 8.82 --dcp-size 8 turns on decode context parallelism for the 24 MLA layers: each token’s cache entry is stored on one of the 8 GPUs instead of on all, which I will call a striped cache. The price is one extra exchange between the GPUs in every MLA layer of a decode step. The KDA layers are not affected, because their state has nothing to stripe. With the cache costing an eighth per token, the calculator moves most of the memory to the state pool, which is why the ratio jumps from 1.1 to 8.82. On our server, the Balanced recipe admitted 101 requests instead of 60 and held 936,448 tokens instead of 561,728. On top of Balanced, the cookbook offers speculative decoding with a drafter, a small model trained for K3 DSPARK . It proposes 7 tokens, and K3 verifies them in one pass: --speculative-algorithm DSPARK --speculative-draft-model-path RadixArk/Kimi-K3-DSpark \ --speculative-dspark-block-size 7 --enable-linear-replayssm-spec \ --mamba-full-memory-ratio 6.28 --max-running-requests 256 The drafter keeps its own cache, which is not striped, so the calculator’s ratio changes again, while --enable-linear-replayssm-spec keeps the D of section 1.3 at zero. SGLang’s default with speculation is 48 running requests; the command’s 256 leaves the limit to the state pool. This server admitted 68 requests and held 697,856 tokens. The cookbook reports the same 101 and 68. I ran the agent requests from section 1.4 against all three servers, and Figure 4 shows what the stripe and the drafter cost or gain. Let’s start with a single request, on the left side of Figure 4. Without the drafter, a token takes 7.7 ms with the Low-Latency recipe and 8.5 ms with the striped cache, so the stripe costs about 10%. With the drafter, it takes 3.8 ms, or 260 tokens per second. The first token arrives after about 0.44 seconds in all three. The cookbook pins the accept length tokens gained per verify pass at 4.5 through an environment variable. On our agent requests, sampled at temperature 1.0, the server reported 2.7 in the run for Figure 4’s left side and 3.7 to 3.8 in the two runs for its right side. I haven’t looked into the difference. With 16 or 64 requests at once, the recipes move close together, and at 64 the server completes about 2 requests per second. At 8,400 tokens each, that is roughly 17,000 prompt tokens per second. For this traffic, a short answer after a long prompt, the server is mostly limited by prefill, and the drafter adds 12% at 64 requests. For the right side of Figure 4, I started each server again and let requests arrive for 4 minutes per point. At one request every 2 seconds, 99% of the requests meet the target on the Low-Latency server and 96% on the Balanced one. At 0.7 requests per second, it is 93% and 91%, and at 0.95, only 75% and 72%. With the drafter, all requests meet it at one every 2 seconds, 94% at 0.7, 93% at 0.95, and 86% still at 1.2. Read strictly, the sustained rate is between 0.5 and 0.7 requests per second for all three recipes. What the drafter changes is how the server degrades above that: it stays within 2 points of 95% up to almost 1. Without the drafter, from 0.95 up, most misses are streams below 40 tokens per second. The drafter removes most of those, and what is left is split between slow streams and first tokens that waited behind other prompts. Each point is one run, so I wouldn’t read much into a difference of 2 or 3 points. But the gap between the 2 requests per second the server can process and what it serves within the target is clear. Finally, I started 4 sessions at once, each with a history of about 118,000 tokens, and then sent 3 returning turns per session. The first pass took 24.2 seconds, a calculated 19,400 tokens per second. Then a returning turn got its first token after 0.45 seconds and ran at 11 to 15 ms per token, with or without the stripe. With 8 sessions on the Balanced server, about 970,000 tokens, more than its cache pool holds, the returning turns took 6 seconds to the first token, presumably because the server kept evicting one session to make room for another. This is the bet from section 1.3 at work. At the calculator’s ratio for requests of 9,216 tokens, the Balanced server holds only 7 long sessions, because most of what the stripe freed went to the state pool. For a server that mainly holds long sessions, I would compute the ratio for their length 0.63 for 128,000 tokens, 0.45 with the drafter and accept that fewer requests can run at once. Which recipe would I launch? Balanced with the drafter. The stripe costs a few points below 1 request per second 10 at 1.2 , and only the drafter raises the share of requests within the target. I would set the ratio from the traffic and watch the accept length. Memory is tight: the Balanced server peaked at 259 GiB of 268.6, and at 263 with the drafter. 2.4 Why Not Split Prefill and Decoding Here 24-why-not-split-prefill-and-decoding-here At rack scale, prefill and decoding usually get separate servers. Here, that’s not an option: two 4-GPU servers can’t each hold a copy of the weights. What one server pays for doing both is still worth measuring, because that’s the cost a split removes. I let 16 requests decode and, after a quiet period, sent one prompt per second for 24 seconds, each asking for one token. Figure 5 shows what happened to the 16 streams. In Figure 5, with prompts of about 8,400 tokens, the streams lost 39% of their output during the window, and 31% in a second run that isn’t drawn, so roughly a third. With prompts cut to about 2,000 tokens, the size a returning turn adds, they lost 15%, in one run. This partly explains the right side of Figure 4. At one request per second, every decoding stream shares the GPUs with a prompt about 40% of the time, since each took 0.42 seconds to prefill. 2.5 Summary 25-summary To summarize, on 8 B300 GPUs the weights decide the layout: one tensor-parallel server whose bar shows 195 GiB of weights and 30 GiB in the two pools. A striped cache makes a cache entry 8 times cheaper at 10% more time per token for a single request, and a drafter halves that time. With long prompts and short answers, the server is mostly limited by prefill, near 2 requests per second. Its sustained rate is between 0.5 and 0.7 requests per second, and with the drafter it stays within 2 points of 95% up to almost 1. This size can’t split prefill from decoding, which costs the decoding streams about a third of their throughput at one fresh prompt per second. 3. Thirty-Two H200 GPUs: One Wide Expert-Parallel Domain 3-thirty-two-h200-gpus-one-wide-expert-parallel-domain Let’s say we have four servers instead of one. The obvious thing would be to launch the server from section 2 four times. On smaller GPUs, that’s not possible, and people run a different shape: one SGLang server across all 32 GPUs, with the experts spread 32 ways. I will call the GPUs that share one copy of the experts a domain. The cookbook mentions a 64-GPU sweep, but 32 GPUs is the widest K3 run I found posted with a recipe. 3.1 Why Wider 31-why-wider An H200 is a Hopper GPU with a nominal 141 GB of memory, which I take as decimal: 131 GiB. If we redo the bar from section 2.1 for it, four copies of the 8-GPU server are out, since each GPU would need 192 GiB. With 16 GPUs, the weights still take up 102 GiB per GPU, which fits but leaves little for the two pools. The cookbook’s basic H200 recipe is 16 GPUs wide. With 32 GPUs, each GPU holds 57 GiB of weights, as Figure 6 shows, and almost half the card is left for state and cache. The width in Figure 6 comes from the weights. To cut the experts 32 ways, we can give every GPU a slice of every expert, as tensor parallelism does, or 28 whole experts, which is expert parallelism. Both cuts cost the same memory, and a decode step reads the same weights either way. The cookbook’s recipe for this size uses whole experts, and so do I. 3.2 The Command 32-the-command Below is the command I use on four servers with 8 H200 GPUs each. It selects no dispatch backend to send tokens to their experts, and section 3.3 says why: python -m sglang.launch server \ --model-path moonshotai/Kimi-K3 \ --tokenizer-path moonshotai/Kimi-K3 \ --served-model-name kimi-k3 \ --tensor-parallel-size 32 \ --ep-size 32 \ --nnodes 4 \ --node-rank "$NODE RANK" \ --dist-init-addr "$MASTER ADDR:20000" \ --dist-timeout 3600 \ --kv-cache-dtype fp8 e4m3 \ --mem-fraction-static 0.88 \ --max-running-requests 64 \ --mamba-full-memory-ratio 0.08 \ --mamba-radix-cache-strategy extra buffer lazy \ --decode-attention-backend flashmla \ --moe-runner-backend marlin \ --reasoning-parser kimi k3 \ --tool-call-parser kimi k3 \ --trust-remote-code \ --enable-metrics --kv-cache-dtype fp8 e4m3 stores the MLA cache in 8 bits and halves the cost of a token to 13,824 bytes. This matters more here than on 8 GPUs, since --tensor-parallel-size 32 makes attention 32 GPUs wide, and tensor parallelism stores the MLA cache on each of them. A session with 128,000 tokens costs 1.6 GiB on each of the 32 GPUs. The state gets cheaper with the width, though. With 96 heads on 32 GPUs, each GPU computes 3 heads, so a state block shrinks from 53.6 MiB to 13.4 MiB. --mamba-radix-cache-strategy extra buffer lazy also lowers the number of blocks per request from 5 to 4 by creating one of the copies only when needed. Both shrink the state’s share, so --mamba-full-memory-ratio is as low as 0.08. In the formula from section 1.3, this ratio corresponds to requests of about 51,000 tokens. By my arithmetic, the cache pool holds about 80 requests of that length, or 33 of our 128,000-token sessions. As a rough check, the cookbook’s run of this shape, with the drafter, a 16-bit cache, and a 10% reserve, had room for 1,940,352 tokens. That is 50 GiB per GPU before the drafter’s cache, next to the 55 GiB cache pool in Figure 6. --max-running-requests 64 caps the batch at a number I chose. The state pool would admit about 80. --mem-fraction-static 0.88 keeps 12% of each GPU outside the pools. Most remaining flags follow the cookbook’s High-Throughput recipe for H200, which has the same shape. flashmla and marlin are the kernels it uses on Hopper for MLA decoding and for the 4-bit experts. 3.3 What the Servers Exchange 33-what-the-servers-exchange With expert parallelism, a dispatch backend usually sends each token to the GPUs that hold its experts. Here, that step isn’t needed. Since attention is tensor-parallel over all 32 GPUs, its partial results are summed across all of them. Afterward, every GPU holds the hidden states of every token. Each GPU then runs its own 28 experts on the tokens routed to them, and the outputs are summed across the GPUs again. In every layer, the four servers exchange two sums, one after the attention module and one after the MoE module, as shown at the bottom of Figure 6. The second also covers the 2 shared experts, which every token uses and which are still sliced 32 ways. With 93 layers, that’s 186 sums per decode step across the network. They are plain sums, since the fused ones from section 2.1 aren’t enabled on Hopper by default. In a decode step with 64 requests, I calculate that one sum carries about 1 MB, so the cost is the latency of 186 sums and not their bandwidth. During prefill, one sum for a chunk of 8,192 tokens the default on this card carries more than 100 MB by the same calculation, and that’s where I would expect this shape to hurt. Why not keep attention in four groups of 8, one per server, and let only the experts span the servers, as section 4.4 does? A GPU then holds only its own group’s tokens. To reach experts on other GPUs, tokens need a dispatch backend, which by default copies the shared experts to every GPU, or the whole batch has to be copied to every GPU in every layer. On H200, with a plain network between servers, I use the simpler shape. 3.4 What Changes on Blackwell 34-what-changes-on-blackwell On B300 and GB300 GPUs, a different MoE path becomes attractive. MegaMoE is a fused kernel that both sends tokens to their experts and runs them. The cookbook’s presets for 16 to 64 Blackwell GPUs are built on it, together with DP attention. There is a catch at 32 GPUs. The kernel reaches the other GPUs through memory mapped over NVLink, so all the GPUs that share the experts have to be connected by NVLink, in one NVLink domain. An open issue https://github.com/sgl-project/sglang/issues/36303 documents what happens otherwise on Hopper. Across two H200 servers, the kernel fails at the first token, and nothing warns about it at startup. A B300 server is also its own NVLink domain with 8 GPUs, so I would expect the same for four of them, though the report covers only Hopper. The cookbook implies the opposite, because it prints its preset for four B200 or B300 servers. I couldn’t find a posted run of it. On a GB300 system, where the NVLink domain spans many nodes, a 32-GPU domain with the fused kernel is possible, and there is one public report of it in production-like use. Unfortunately, it’s a bug report https://github.com/sgl-project/sglang/issues/40751 . On a coding benchmark, K3 solved 68% of the tasks with tensor parallelism over 8 GPUs, about 65.5% with expert parallelism over 32 GPUs and an unfused backend, and 42% with the fused kernel over 32 GPUs. The fused run was a decode server with attention in eight groups of 4, unlike section 4.4’s four groups of 8, and half its outputs ran into long repetitions. In that build, the loss follows the fused kernel, not the width, since the unfused run at the same width was close to the baseline. The reporter hasn’t found the cause. 3.5 Summary 35-summary To summarize, the answer at 32 GPUs depends on the hardware. On H200, memory forces the width, which shrinks the weights in the bar to 57 GiB, and the price is two sums per layer across the network. On four B300 servers, I would run the server from section 2 four times, because the fused MoE kernel probably can’t cross between servers. The wide shape becomes interesting on Blackwell only once the NVLink domain is larger than a server. 4. A GB300 Rack: A Prefill Pool and a Decode Pool 4-a-gb300-rack-a-prefill-pool-and-a-decode-pool A GB300 NVL72 https://www.nvidia.com/en-us/data-center/gb300-nvl72/ has 72 GPUs and 20 TB of GPU memory, in nodes of 4 GPUs, and all of it is one NVLink domain. For K3, this means room for nine groups of 8 GPUs, each with its own copy of the weights. The limit from section 3.4 is gone too, since the fused MoE kernel can reach any GPU in the rack. 4.1 Three Ways to Fill a Rack 41-three-ways-to-fill-a-rack With nine groups, we can finally give prefill and decoding their own GPUs prefill-decode disaggregation , which should remove the cost we measured in section 2.4. I see three ways to use the rack: - a Nine unified servers: the server from section 2, nine times, behind a load balancer. This is the baseline. - b A prefill pool and independent 8-GPU decode servers: some groups only prefill, the others only decode, with a router in front and the request’s state transferred in between. This layout has measurements behind it. - c A prefill pool and one or two wide decode domains: the same split, but the decode side is a single server over 32 or 64 GPUs with the experts spread across all of them. This is a proposal. The K3 recipes of the open InferenceX https://github.com/SemiAnalysisAI/InferenceX benchmark are for other engines and stop at 32 GPUs: a prefill worker and three 8-GPU decode servers, which is layout b . 4.2 The Prefill Pool 42-the-prefill-pool Layouts b and c share the prefill side. For this role, the cookbook’s Long-Context recipe is what it calls deep pipeline parallelism: no tensor parallelism and one pipeline stage per GPU. With 8 GPUs, each stage holds 11 or 12 of the 93 layers in full, and a prompt moves through the stages chunk by chunk. prefill worker: 8 GPUs on two nodes from the cookbook's GB300 Long-Context prefill recipe sglang serve --model-path moonshotai/Kimi-K3 --trust-remote-code \ --tp-size 1 --pp-size 8 --nnodes 2 --node-rank