Scaling Kimi K3 on AMD Instinct MI355X: Matching NVIDIA HGX B300 Throughput per GPU on AgentX

An eight-GPU AMD Instinct MI355X system serving Kimi K3 with vLLM delivers 24.4% more total tokens per second per GPU than NVIDIA HGX B300 at 50 tokens/s/user P90 interactivity on the AgentX benchmark, according to AMD and vLLM community collaborators. At that interactivity level, the MI355X system serves approximately 7,567 total tokens/s/GPU versus 6,083 for NVIDIA HGX B300, and at SemiAnalysis's modeled TCO of $1.50 versus $2.26 per GPU-hour it yields 18.16 million versus 9.69 million tokens per dollar, 87.4% more. The result rests on vLLM serving changes including eight-way decode context parallelism (DCP8), which stores one eighth of cached positions per GPU, and ROCm AITER DCP support for multi-token verification added in vLLM PR #51705.

Scaling Kimi K3 on AMD Instinct MI355X: Matching NVIDIA HGX B300 Throughput per GPU on AgentX Inside the Kimi K3 optimization effort: understanding the bottlenecks and the decisions that made inference faster On AgentX, an eight-GPU AMD Instinct MI355X system serving Kimi K3 with vLLM delivers 24.4% more total tokens per second per GPU than NVIDIA HGX B300 at 50 tokens/s/user P90 interactivity. Under the benchmark’s ownership-cost model, it delivers 87.4% more tokens per dollar . In this post, we’ll unpack the engineering behind Kimi K3 on AMD Instinct MI355X, connecting vLLM’s serving decisions to GPU kernels and the hardware underneath. We’ll explain why the optimizations worked and build the intuition to spot wasted work, identify bottlenecks and find the next optimization in your own inference stack. Published dense matrix peaks are 10.1 PFLOP/s MXFP4 for AMD Instinct MI355X and 13.5 PFLOP/s FP4 for NVIDIA B300. Both have 288 GB HBM3E and 8 TB/s peak memory bandwidth. Reaching this serving performance reflects sustained software engineering by AMD and collaborators across the vLLM community. AgentX: the workload and results The public AgentX https://vllm.ai/blog/2026-09-08-vllm-agentx benchmark replays real coding-agent session traces with synthetic tokens, retaining request lengths, timing and prefix-sharing patterns. The workload offers over 96% theoretical prefix reuse with unlimited cache. Speculative runs use a fixed synthetic acceptance reference. At 50 tokens/s/user P90 interactivity , AMD Instinct MI355X serves approximately 7,567 total tokens/s/GPU , compared with 6,083 for NVIDIA HGX B300 , a 24.4% throughput advantage . Using SemiAnalysis’s modeled TCO of $1.50 versus $2.26 per GPU-hour , respectively, that translates to 18.16 million versus 9.69 million total tokens per dollar of TCO: 87.4% more tokens per dollar . The public curves compare complete serving stacks. The measurements beside each optimization come from separate tests and explain individual mechanisms under their stated configurations. Inside Kimi K3: experts, attention and recurrent state A returning agent turn can reuse most of its prefix, but processing a short continuation still touches three kinds of data. Kimi K3 uses two kinds of attention. Its 24 MLA layers retain a compressed KV history that grows with the conversation. A new query still consults that history. Its 69 KDA layers instead carry forward state matrices with fixed dimensions, updating them as new tokens arrive. Model configuration https://huggingface.co/moonshotai/Kimi-K3/blob/c5d1dd4c428bd1ce8b88c5044f3b6ccde9e3b721/config.json . In MoE layers, a router selects expert networks to transform each token. Their weights are shared across requests. Across turns: cache reuse and offload Section 1 . Within a model step: expert projections, kernel handoffs, and MLA/KDA scheduling Sections 2–5 . Choosing draft depth: committed output versus complete cycle cost Section 6 . 1. Cache: keep conversation state available for reuse DCP reduces duplicate history in GPU memory; CPU offload preserves reusable state outside it. DCP: share the history across GPUs Under TP8, each GPU handles 12 of Kimi K3’s 96 MLA query heads but stores the entire compressed KV history. Tensor parallelism divides the heads without dividing the cache. With eight-way decode context parallelism DCP8 https://vllm.ai/blog/2026-08-07-decode-context-parallelism , each GPU stores one eighth of the cached positions. During decode, the GPUs exchange queries, compute against their local history and merge their partial answers. Each partial answer carries normalization information so the merge can weight shards correctly. 51705 https://github.com/vllm-project/vllm/pull/51705 extends the ROCm AITER DCP path to multi-token verification. For a 142,000-token conversation, the estimated MLA payload is 24 layers × 142,000 positions × 576 FP8 bytes ≈ 1.96 GB per GPU. DCP8 reduces this to about 245 MB, before allocation overhead. Each GPU processes eight times as many heads against one eighth of the history, leaving the leading attention arithmetic roughly unchanged. CPU offload: keep state that no longer fits in HBM When state no longer fits in GPU memory, the AMD recipe https://github.com/SemiAnalysisAI/InferenceX/blob/8adb5c5589b84c0afdccc60bab521eb1114f52b5/inferencex-e2e/benchmarks/single node/srt-slurm-recipes/kimik3/vllm/mi355x-fp4-mtp/agentic.yaml ’s native vLLM CPU offload preserves MLA history and KDA checkpoints in host DRAM. A cache hit needs matching resume points. If MLA history reaches token 3,072 but the latest compatible KDA checkpoint is at 2,048, the request must resume at 2,048 and recompute the remaining prefix. Saving the attention-block tail alongside the KDA checkpoint aligns their resume positions 54736 https://github.com/vllm-project/vllm/pull/54736 . Restoring that state trades a transfer back to the GPU for avoided prefix recomputation. Once a request resumes, its new tokens can still make several GPUs repeat the same calculation. 2. MoE: remove duplicate work across GPUs On one GPU, the routed-expert output projection returns the complete vector its consumer expects. In the original ROCm path, an earlier reduction and normalization gave all eight GPUs the same latent input, and each held the full weights. The same projection therefore ran eight times. The output coordinates can be divided among GPUs, but every rank still needs the complete result. Kimi K3 already has an all-reduce for its shared-expert partials. Each GPU adds only the routed outputs it owns to its shared-expert partial. The existing all-reduce assembles the complete result, with no new collective 51253 https://github.com/vllm-project/vllm/pull/51253 . The ROCm implementation builds on the corresponding CUDA strategy. Each output coordinate receives its routed contribution exactly once. The full projection weights remain replicated. Output throughput rose 3.05–4.91% in the PR’s test on eight AMD Instinct MI355X GPUs at TP8, across concurrency 8–128 with random 8K-input/1K-output requests and prefix caching disabled. Free up GPU memory to keep more cached state Each rank now computes only its assigned projection coordinates while retaining the full weights. Loading only the rows each rank uses frees about 3.85 GiB per GPU across 92 MoE layers at TP8 59591 https://github.com/vllm-project/vllm/pull/59591 . The fused path’s calculation stays the same; the extra space can hold MLA history and KDA checkpoints. This change landed after the image used for the headline public result. Its separate AgentX experiment found less than 1% higher throughput at concurrency 32 and 48, but 28.9% at 70 , where the reported prefix-cache hit rate rose from 74.8% to 80.1%. The larger gain at high concurrency is consistent with cache pressure. Comparing both implementations at equal cache capacity would help isolate that effect. More on expert routing and four-bit activations technical-note-expert-routing-and-precision . 3. Memory traffic: fuse operations and remove unnecessary copies Within each GPU, the next place to look is the data passed between kernels. Kimi K3’s single-token KDA path ran three compute kernels and two output copies. Convolution ran before a kernel that already fused Q/K normalization, decay and beta gates, and the recurrent update; output normalization ran afterward. The fused HIP path performs these operations in one launch and writes directly to the destination buffer 50654 https://github.com/vllm-project/vllm/pull/50654 . Intermediate handoffs stay on chip. Convolution history and recurrent state still go to GPU memory for the next token. AttnRes mixes earlier layer-block outputs with the running residual. Combining that mix with RMSNorm keeps the temporary mixture in registers 50593 https://github.com/vllm-project/vllm/pull/50593 . The running residual saved for later layers retains its BF16 rounding; the mixture stays in FP32 through normalization. Tests require exact persistent updates and allow numerical tolerances for normalized output. With AttnRes fusion, output throughput rose 14.2% on eight AMD Instinct MI355X GPUs at TP8, with 100K-input/1K-output requests and prefix caching disabled. Long-prefill latency was effectively unchanged. Copy inputs only for kernels that use them The ROCm matrix-multiplication dispatcher packed inputs before choosing a kernel. When it selected the fallback, that kernel read the original input and left the copy unused. Moving packing into the specialized branches removed these copies 58566 https://github.com/vllm-project/vllm/pull/58566 . The profile made the waste visible: 69 unnecessary copies per decode step , one per KDA layer. Each moved only 1,792 bytes but took about 4.8 microseconds. Moving those bytes faster would barely help; launch overhead dominated. 4. MLA: faster long-context decoding with AMD AITER assembly Kimi K3 has twelve query heads per GPU at TP8. Padding to sixteen made AMD AITER’s assembly decode path available. In the reported long-context test, time per output token fell from 43.8 to 23.6 milliseconds. The comparison used TP8 without DCP, at concurrency one. The adapter in 50578 https://github.com/vllm-project/vllm/pull/50578 copies four query heads, packs them into the sixteen-head layout and discards their outputs. DCP8 presents a different, 96-head shape. This preserves the model’s operation because each query head attends independently over cached positions. Adding another head does not change the original head’s softmax. Only the query is padded in this decode path; the stored compressed KV history stays the same size. A long conversation gives attention plenty of work, but that work needs to be spread across the GPU. In the historical code we inspected, a conservative scheduling hint https://github.com/vllm-project/vllm/blob/7b9f2dad8920f115c1caea36e096e43c04c3da68/vllm/v1/attention/backends/mla/rocm aiter mla.py L1092 restricted the Gluon path to one workgroup per request https://github.com/ROCm/aiter/blob/31350226161346314b3d8882c8085bd31dce6a34/aiter/ops/triton/gluon/mla gluon.py L937 . That workgroup still processed the entire conversation. Padding made AMD AITER’s assembly path available, with a different policy https://github.com/ROCm/aiter/blob/31350226161346314b3d8882c8085bd31dce6a34/aiter/mla.py L149 for dividing the history into parallel pieces and combining the results. | Input / output tokens | Original Gluon path | Padded assembly path | |---|---|---| | 1K / 1K | 22.9 ms/token | 22.8 ms/token | | 32K / 1K | 43.8 ms/token | 23.6 ms/token | PR-reported mean time per output token: eight AMD Instinct MI355X GPUs, TP8, concurrency one, MXFP4 model weights, BF16 attention and ordinary single-token decode. The short-input case barely changes; the long-input case nearly halves time per output token. These serving measurements include query preparation and output handling. 5. KDA: parallelize prefill and speculative verification MLA can split a retained history among workgroups. KDA presents a harder scheduling problem: each token depends on the preceding state. Fused prefill in 54038 https://github.com/vllm-project/vllm/pull/54038 prepares each token group’s combined transformation before its incoming state is known. For example, “halve the state and add two,” then “halve it and add one,” combine into “take a quarter of the starting state and add two.” This rule can be computed before the starting state arrives. Step through the KDA dependency sequence KDA does this with matrix transformations. An ordered scan of the group summaries determines each group’s starting state; the groups then replay their tokens in parallel. Token order stays intact, but each group no longer waits for the preceding group’s replay. Grouping adds a second walk and expands the fused chunk path from two launches to four. It helps when few sequences or heads expose too little parallel work; otherwise, the extra passes can cost more than they save. Within each group, the recurrent walk keeps state in matrix-accumulator registers between chunks, avoiding a global-memory write and reload. State storage diagram. state-between-chunks Fused KDA delivered 5.3–5.6% more input throughput than Triton at 4K and 16K input lengths in a separate three-backend comparison: eight AMD Instinct MI355X GPUs, TP8, concurrency 32, prefix caching disabled and one-token outputs. This measures the complete backend. Benchmark results https://github.com/vllm-project/vllm/pull/56036 issuecomment-5667437134 . Use smaller tiles for speculative verification Short verification batches need a different split. At TP8, each GPU has 12 KDA heads with 128 value columns. Splitting each head into four 32-column tiles launches only 48 workgroups on 256 compute units. 58045 https://github.com/vllm-project/vllm/pull/58045 switches to four-column tiles, raising the count to 384. In a separate graph-replay kernel test on one AMD Instinct MI355X GPU TP8 dimensions, one sequence, four verification positions , time fell from 16.20 to 5.61 microseconds per layer 2.89× . This includes narrower tiles, single-wave workgroups, uniform-length specialization and hoisted token-invariant work. Ragged batches retain a dynamic path; tests compare outputs and recurrent state. 6. Speculative decoding: choose how many tokens to draft Faster verification can change how many draft tokens are worth checking. The useful rate is committed output divided by the full draft–verify–commit cycle time. For example, AgentX’s fixed acceptance reference gives 3.00 expected committed tokens per cycle with three drafts, and 3.36 with four, including the recovery or bonus token. That is 12% more committed output per cycle. Four drafts improve throughput only if the complete cycle takes less than 12% longer than three drafts. The public AMD recipe https://github.com/SemiAnalysisAI/InferenceX/blob/8adb5c5589b84c0afdccc60bab521eb1114f52b5/inferencex-e2e/benchmarks/single node/srt-slurm-recipes/kimik3/vllm/mi355x-fp4-mtp/agentic.yaml uses three to seven drafts with DCP1 and disables speculation at the larger DCP8 concurrency settings. What to optimize next Ongoing prefill/decode disaggregation work adds another place to look: worker balance and whether transfers finish before their data is needed. For another model or workload, start with the deployed request shapes and make three checks: Compare ranks. Are several GPUs computing the same result before a collective that could assemble it? Follow buffers. Does each temporary tensor have a consumer on the selected path? Can the data stay on chip? Count independent work. Do the actual heads, tokens and tiles produce enough workgroups to spread the load? Measure the complete request at the response speed you need, and check both outputs and saved state. A faster kernel matters when it shortens the work holding up the next token. Contributions and acknowledgments The Kimi K3 effort spans more than 50 merged AMD/ROCm-related vLLM PRs from AMD, Embedded LLM and the wider community, alongside AITER development, integration, review and benchmarking. Contributions cover attention, recurrence, MoE, quantization, graph execution, cache support and platform integration. Special thanks to: Pin Siang Tan, Jun Kang Chow, Ye Hur Cheong, Hongxia Yang, Seungrok Jung, Bill He, Yichao Zhu, Hyukjoon Lee, Andy Luo, Mrinal Karvir, Richard Li, Sun Peng Andy Friedrich https://github.com/afriedri , Ethan Yang https://github.com/amd-ethany , Tang Rui https://github.com/amd-ruitang3 , Andreas Karatzas https://github.com/AndreasKaratzas , Ankur Singh https://github.com/Ankur-singh , Yihua Cheng https://github.com/ApostaC , Badr Ba Sowid https://github.com/BadrBasowid , Benjamin Chislett https://github.com/benchislett , Taoyu Zhu https://github.com/benenzhu , Bill He https://github.com/billishyahao , Bin Ding https://github.com/binding7012 , Bowen Bao https://github.com/BowenBao , Brandon Zhang https://github.com/brandonfzhang , Chaemin Lim https://github.com/chaeminlim-mb , Charles Wu https://github.com/charxwu , Chuan Richard Li https://github.com/ChuanLi1101 , Felix Li https://github.com/coderfeli , Cameron Quilici https://github.com/cquil11 , Weishu Deng https://github.com/deng451e , Dewei Wang https://github.com/Dewei-Wang-sh , Divakar Verma https://github.com/divakar-amd , Douglas Lehr https://github.com/dllehr-amd , Edwin Lim https://github.com/edwinlim0919 , Yudong Pan https://github.com/eecspan , Elvir Crnčević https://github.com/elvircrn , Fang Che https://github.com/fangche123 , Fangzhou Ai https://github.com/Fangzhou-Ai , Xingran Fan https://github.com/fanxingran , Fulu Li https://github.com/fululi12 , Felix Marty https://github.com/fxmarty-amd , Canlin Guo https://github.com/gcanlin , Summer Yang https://github.com/GirasoleY , Haichen Zhang https://github.com/haic0 , Yufeng He https://github.com/he-yufeng , He Liu https://github.com/heliubj18 , Hongxia Yang https://github.com/hongxiayang , Hyukjoon Lee https://github.com/hyukjlee , Yifan Qiao https://github.com/ivanium , James E T Smith https://github.com/jamesETsmith , Jiahui Cao https://github.com/jiacao-amd , Yanyuan Qin https://github.com/JohnQinAMD , Jonathan Chen https://github.com/jonathanc-n , Juntian Liu https://github.com/Juntian777 , Shao-Chun Lee https://github.com/k50112113 , Kedar Potdar https://github.com/kedarpotdar-nv , Kevin H. Luu https://github.com/khluu , Kuan-Fu Liu https://github.com/kliuae , Kuntai Du https://github.com/KuntaiDu , Kyle Sayers https://github.com/kylesayrs , @lalala-sh https://github.com/lalala-sh , @linnea-lin-00638949 https://github.com/linnea-lin-00638949 , Yinfeng Liu https://github.com/LiuYinfeng01 , Lucas Wilkinson https://github.com/LucasWilkinson , Gabriel Wu https://github.com/lucifer1004 , Markus Hartikainen https://github.com/maeehart , Maithili Joshi https://github.com/maithilijoshi20 , Jeff Junze Ma https://github.com/majunze2001 , Matthew Wong https://github.com/mawong-amd , Misha Goin https://github.com/mgoin , Micah Williamson https://github.com/micah-wil , @minmengdie https://github.com/minmengdie , Matvei Pashkovskii https://github.com/mpashkovskii , Mathews Neha https://github.com/nehmathe2 , Nicolò Lucchesi https://github.com/NickLucche , Nick Hill https://github.com/njhill , Nicholas Susanto https://github.com/nsusanto , Oxana Korzh https://github.com/okorzh-amd , Pei Zhang https://github.com/peizhang56 , @ppalanga https://github.com/ppalanga , Peiran Qin https://github.com/QinPR , Qiuyang Yue https://github.com/qyYue1389 , Rita Brugarolas https://github.com/rbrugaro-amd , Rebecca Lee https://github.com/rebklee , Ryan Rock https://github.com/rjrock , Rohan Potdar https://github.com/Rohan138 , @Saddss https://github.com/Saddss , Samuel Shen https://github.com/sammshen , Sean Filimon https://github.com/seanfilimon , Seungrok Jung https://github.com/seungrokj , Shantipriya Parida https://github.com/shantipriya-amd , Shanshan Shen https://github.com/shen-shanshan , Shengnan Xu https://github.com/shengnxu , Simon Danielsson https://github.com/simondanielsson , Siraj Shaikh https://github.com/Siraj637909 , Stacy Roberts https://github.com/stacyroberts , Stefan Koncarevic https://github.com/stefankoncarevic , Asan Stefanski https://github.com/stefanskiasan , Minghui Jiang https://github.com/Sy0307 , Tarık Saraç https://github.com/tarik-sarac , Tun Jian Tan https://github.com/tjtanaa , Lingpeng Jin https://github.com/valarLip , Vansh Bhatia https://github.com/vanshbhatia-amd , Maksim Vedenev https://github.com/vedenev-amd , Ye Hur Cheong https://github.com/vllmellm , @wenwzhan https://github.com/wenwzhan , Wei Zhao https://github.com/wzhao18 , Xavier Aguilar https://github.com/xaguilar-amd , Xiaobing Zhang https://github.com/XiaobingSuper , Xiaohu Guo https://github.com/xiaohuguo2023 , Xin Li https://github.com/xinli-sw , Xudong Lyu https://github.com/xudonlyu , Yanxue Ren https://github.com/yanxuer-999 , Wentao Ye https://github.com/yewentao256 , Yongqin Wang https://github.com/yongqinwang-cmd , Riggins Chen https://github.com/yoo-kumaneko , Chao Hong https://github.com/youzhedian , Yuchen Shen https://github.com/yucshen , Matt Qin https://github.com/yudigege86 , Yichao Zhu https://github.com/YukioZzz , Roger Wang https://github.com/ywang96 , Zheng Gong https://github.com/ZhengGong-amd , Jiangyun Zhu https://github.com/ZJY0516 , Zvi Grinberg https://github.com/zvigrinberg , Yongye Zhu https://github.com/zyongye . Benchmark notes Serving configurations at the comparison targets | P90 interactivity tokens/s/user | AMD Instinct MI355X | NVIDIA HGX B300 | |---|---|---| | 30 and 50 | SimpleCPU offload; DCP8 / no drafts to DCP1 / three drafts | Mooncake offload; DCP8 | | 150 | GPU-only; DCP1; five to seven drafts | Mooncake offload; DCP8 | Both systems use eight GPUs, TP8, FP4 weights and FP8 KV. Values interpolate within each published offload or GPU-only series at matching interactivity; concurrency, DCP and draft depth can vary between the bracketing runs. The pinned AMD recipe YAML https://github.com/SemiAnalysisAI/InferenceX/blob/8adb5c5589b84c0afdccc60bab521eb1114f52b5/inferencex-e2e/benchmarks/single node/srt-slurm-recipes/kimik3/vllm/mi355x-fp4-mtp/agentic.yaml and NVIDIA launcher https://github.com/SemiAnalysisAI/InferenceX/blob/630dc8c73f84d6f3a74062384c86aa660440929f/benchmarks/single node/agentic/kimik3 fp4 b300 vllm mtp.sh give the image tags and full settings. Replay uses the cc-traces-weka-062126 https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126/blob/23f152f6f0f9399a85901b89a6458def0ef16729/README.md dataset and a pinned AIPerf harness https://github.com/SemiAnalysisAI/agentx-harness/blob/754356e9a39acc6cc6afb242d123bb57c3fb6f75/docs/tutorials/agentx-mvp.md optional-cache-pressure-warmup , with warmup excluded and one hour of profiling under fixed synthetic acceptance curves https://github.com/SemiAnalysisAI/InferenceX/blob/a79e12d2c1fbb86ead753cb7c7d05e11acbb1185/golden al distribution/kimik3 dspark probabilistic sample method block rejection sample method.yaml . The recipes give the seed, session-start and warmup settings. The downloaded dataset revision is unrecorded. Closed-loop replay can produce different completed request mixes. Software and offload. The submissions use vLLM 36768d1bf https://github.com/vllm-project/vllm/tree/36768d1bfd39094681cdbc8cb37d4b31c0729c89 AMD and 5894fdf98 NVIDIA . Configured host-cache budgets are 224.875 and 281 GB per GPU, respectively; connector and image settings are in the linked recipes. Costs. SemiAnalysis’s large-hyperscaler ownership model https://github.com/SemiAnalysisAI/InferenceX-app/blob/43e626e6baa05af8e7932c85d02a82d70c9e68cf/packages/constants/src/tco.ts and published GPU cost constants https://github.com/SemiAnalysisAI/InferenceX-app/blob/43e626e6baa05af8e7932c85d02a82d70c9e68cf/packages/constants/src/gpu-keys.ts use $1.50 per GPU-hour for AMD Instinct MI355X and $2.26 for NVIDIA HGX B300. Total tokens/$ = total tokens/s/GPU × 3,600 ÷ modeled $/GPU-hour. Some capital and utilization assumptions remain unpublished. At the 50-token interactivity target, the comparison gives 46.6% lower modeled cost per token. Snapshot. SemiAnalysis InferenceX data https://inferencex.semianalysis.com/api/v1/benchmarks?model=Kimi-K3 retrieved October 9, 2026; submissions dated September 30, 2026 AMD and August 16, 2026 NVIDIA . PR benchmarks use their stated workloads; 59591 https://github.com/vllm-project/vllm/pull/59591 is a separate experiment after the headline image. | Published specification, per GPU | AMD Instinct MI355X | NVIDIA B300 GPU | |---|---|---| | Dense low-precision matrix peak | 10.1 PFLOP/s MXFP4 | 13.5 PFLOP/s FP4 | | GPU memory | 288 GB HBM3E | 288 GB HBM3E | | Peak memory bandwidth | 8 TB/s | 8 TB/s | Sources: AMD Instinct MI355X https://www.amd.com/en/products/accelerators/instinct/mi350/mi355x.html , NVIDIA HGX B300 https://www.nvidia.com/en-us/data-center/hgx/ , NVIDIA memory specifications https://docs.nvidia.com/enterprise-reference-architectures/hgx-ai-factory/latest/components.html . Benchmark details: AgentX methodology https://inferencex.semianalysis.com/agentx . Technical note: expert routing and precision Kimi K3 selects 16 of 896 experts per token, so a substantial server batch can still leave few rows per expert. The resulting expert tiles may contain padding. A4W4 reduces a different part of the traffic: the integration in 53940 https://github.com/vllm-project/vllm/pull/53940 changes activations from eight bits to four, while weights remain four-bit. Total traffic also includes BF16 intermediate outputs and expert-tile padding. Technical note: KDA precision and on-chip memory Each chunk still needs to resolve dependencies between its tokens. The kernel combines them into a transformation that later matrix products can apply. That creates two practical questions: which precision preserves the answer, and how much temporary data fits on chip? A small example shows the idea. Substitute the earlier corrections into the later ones, and each answer can be expressed directly in terms of the known inputs. The resulting coefficient table has already accounted for the chain of dependencies: The real prologue computes the inverse of the 64×64 within-chunk dependency matrix. Later matrix products use this transformation to account for those dependencies. The inverse’s intermediate values can grow and cancel. Rounding them too early loses information the final result needs. The implementation uses AMD’s native FP32 matrix instructions https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/instruction-set-architectures/amd-instinct-cdna4-instruction-set-architecture.pdf page=295 , retaining FP32 inputs and accumulation for the sensitive inverse. Only the completed inverse is rounded to BF16 for subsequent products. Kernel implementation https://github.com/vllm-project/vllm/blob/af1c01499b289be555c475669ba50a88e96d846e/csrc/libtorch stable/kimi k3/fused kda chunk kernel rocm.cu L1639 . Why FP32 accumulation cannot recover rounded inputs Consider subtracting 256 from 256.5. Both inputs are exact in FP32. At this magnitude, converting 256.5 to BF16 rounds it to 256: | Input precision | Subtraction performed in FP32 | Result | |---|---|---| | Keep the original FP32 inputs | 256.5 − 256 | 0.5 | | Round the inputs to BF16 first | 256 − 256 | 0 | FP32 subtraction cannot recover information lost before it starts. These toy values illustrate why the inverse uses native 16×16×4 FP32 matrix instructions, retaining both input and accumulation precision. An FP4 model also relies on operations at higher precision. Their performance depends on making those numerical paths efficient. The prologue also has a local-memory budget. AMD calls its on-chip shared scratch LDS. Q and K tiles reuse storage because they are needed at different times; the triangular input stores only its ten occupied 16×16 blocks. The workgroup requests 79.5 KiB , which rounds to 80 KiB , half of an AMD CDNA 4 compute unit’s 160 KiB LDS budget https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/instruction-set-architectures/amd-instinct-cdna4-instruction-set-architecture.pdf page=14 . Test this threshold by reserving enough unused LDS to cross it while keeping the kernel binary fixed. If runtime barely changes, the extra resident workgroup was contributing little for that shape. State between chunks Disclaimers The information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities that cannot be completely prevented or mitigated. AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED ‘AS IS.” AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. AMD, the AMD Arrow logo, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies. © 2026 Advanced Micro Devices, Inc. All rights reserved.