{"slug": "serve-kimi-k2-5-mxfp4-on-mi355x-with-atom", "title": "Serve Kimi-K2.5-MXFP4 on MI355X with ATOM", "summary": "AMD shows how to serve the pre-quantized amd/Kimi-K2.5-MXFP4 checkpoint on AMD Instinct MI355X GPUs using ATOM, a lightweight vLLM-like framework that integrates AITER kernels and exposes an OpenAI-compatible endpoint. ATOM reads MXFP4 metadata from the HuggingFace checkpoint, dispatches optimized gfx950 FP4 kernels, and supports tensor parallelism for large MoE models, with a standalone server mode and a vLLM out-of-tree plugin path for production deployments.", "body_md": "# Serve Kimi-K2.5-MXFP4 on MI355X with ATOM[#](#serve-kimi-k2-5-mxfp4-on-mi355x-with-atom)\n\nIn our previous Kimi-K2.5 posts, we moved from [fused MoE optimization with FlyDSL](https://rocm.blogs.amd.com/artificial-intelligence/kimi-k2.5-optimize/README.html) [[8]](#references) to [W4A8 and W8A8 quantization with AMD Quark](https://rocm.blogs.amd.com/artificial-intelligence/kimi-k2.5-w4a8/README.html) [[9]](#references). Those posts focused on kernel and quantization work: how to make the dominant MoE path faster, and how to trade off INT4, INT8, and FP8 formats on AMD Instinct™ MI300X and MI325X GPUs.\n\nThis post shifts the focus to the serving stack. We show how to serve the pre-quantized [ amd/Kimi-K2.5-MXFP4](https://huggingface.co/amd/Kimi-K2.5-MXFP4) checkpoint\n\n[[2]](#references)on AMD Instinct MI355X with\n\n[ATOM](https://github.com/ROCm/ATOM)\n\n[[3]](#references), so the runtime can recognize MXFP4 metadata, dispatch optimized gfx950 FP4 kernels, and expose an OpenAI-compatible endpoint without manually wiring every model and kernel component.\n\nATOM connects model-specific execution, [AITER](https://github.com/ROCm/aiter) [[4]](#references) kernels, `torch.compile`\n\n, CUDA graph capture, distributed execution, and serving interfaces into a deployable stack. In this post, we use ATOM’s standalone OpenAI-compatible server for measurement and describe the vLLM out-of-tree plugin path for users who want to keep [vLLM](https://github.com/vllm-project/vllm) [[5]](#references) as their serving framework.\n\n## ATOM: Optimized Serving for AMD GPUs[#](#atom-optimized-serving-for-amd-gpus)\n\nATOM, short for AiTer Optimized Model, is a lightweight vLLM-like implementation focused on integrating and optimizing AMD GPU model execution with AITER kernels. In practice, ATOM provides the framework layer around the optimized kernels:\n\n**ROCm and AITER kernel integration**: ATOM routes model operations to AITER kernels, including ASM, CK, Triton, and FlyDSL-backed paths.** OpenAI-compatible serving**: ATOM provides`/v1/completions`\n\nand`/v1/chat/completions`\n\nendpoints for standalone serving.**Compilation and graph capture**: ATOM supports piecewise`torch.compile`\n\nand CUDA graph capture for low-latency decode.**Multi-GPU execution**: ATOM supports tensor parallelism, data parallelism, and expert-parallel execution paths for large MoE models.** Quantization-aware dispatch**: ATOM reads HuggingFace quantization metadata and maps supported formats, including MXFP4, to runtime quantization and kernel choices.\n\nFor Kimi-K2.5-MXFP4, that last point is key. The checkpoint already contains packed FP4 weights and E8M0 scale tensors. ATOM maps those metadata fields to the `per_1x32`\n\nFP4 runtime path and dispatches AITER gfx950 kernels that execute scaled MFMA instructions.\n\n### Standalone ATOM Server[#](#standalone-atom-server)\n\nATOM can run as its own OpenAI-compatible server. This mode is useful for benchmarking and for validating new model support because the optimized ATOM model implementation and AITER kernels are directly visible in the serving stack.\n\n```\npython3 -m atom.entrypoints.openai_server \\\n    --model /models/Kimi-K2.5-MXFP4 \\\n    --trust-remote-code \\\n    -tp 8 \\\n    --kv_cache_dtype fp8 \\\n    --host 0.0.0.0 \\\n    --server-port 18002\n```\n\n### ATOM as a vLLM Out-of-Tree Plugin Backend[#](#atom-as-a-vllm-out-of-tree-plugin-backend)\n\nFor many deployments, vLLM is the serving framework that users already use in production. ATOM can also work as a vLLM out-of-tree (OOT) plugin backend: ATOM is installed as a separate Python package and plugs into vLLM through vLLM’s official plugin interfaces.\n\nIn that mode, vLLM remains the framework-level serving runtime, while ATOM contributes the AMD-optimized model execution path:\n\nATOM registers a vLLM platform plugin so vLLM can resolve the ATOM platform.\n\nATOM registers selected model wrappers such as\n\n`ATOMForCausalLM`\n\nand`ATOMMoEForCausalLM`\n\n.ATOM supplies AITER-backed attention backends for supported models.\n\nvLLM continues to drive request scheduling and serving, while the hot model execution path runs through ATOM model code and AITER kernels.\n\nThis split matters for adoption. The standalone server is the shortest path to validate a new optimized model stack; the vLLM plugin path lets users keep the vLLM interface and operational model while benefiting from ATOM’s AMD-specific model and kernel optimizations.\n\n## From W4A8 Quark Configs to MXFP4[#](#from-w4a8-quark-configs-to-mxfp4)\n\nThe W4A8 post used [AMD Quark](https://github.com/amd/quark) [[6]](#references) to produce a checkpoint compatible with the INT8 MFMA runtime path on gfx942. That required a different weight layout from W4A16: INT4 weights with per-channel scales, plus dynamic activation quantization at runtime. The reason was structural: the INT8 MFMA inner loop could not apply a different floating-point weight scale every 32 K values, so W4A8 moved weight scales to a per-channel layout that could be applied in the epilogue.\n\nFor MXFP4 on gfx950, the config changes in a different direction. Instead of coarsening the weight scale to make an INT8 epilogue work, MXFP4 keeps a fine-grained 32-value block scale and uses a hardware path that can consume that block scale in the matrix instruction.\n\nAt a high level, the Quark-style configuration changes look like this:\n\nConfig choice |\nW4A8 direction |\nMXFP4 direction |\n|---|---|---|\nWeight dtype |\nINT4 |\nFP4 |\nActivation dtype |\nDynamic 8-bit activation path |\nDynamic FP4 activation path |\nWeight scale granularity |\nPer-channel |\nPer-group, |\nScale format |\nFloating-point scale tensors for INT4/activation dequantization |\nE8M0 block scales |\nObserver style |\nINT4/FP8 quantization specs |\n|\nRuntime target |\nINT8 MFMA with scale application in the epilogue |\ngfx950 scaled FP4 MFMA with payload and scale operands |\n\nFor MXFP4, both weights and input activations use `dtype=fp4`\n\n, `qscheme=per_group`\n\n, `group_size=32`\n\n, and `scale_format=e8m0`\n\n. The weights are stored statically in the checkpoint as packed FP4 payloads plus E8M0 `weight_scale`\n\ntensors; activations are quantized dynamically at runtime into the same 1x32 block-scaled representation.\n\nThat makes the checkpoint and runtime contract different from W4A8. W4A8 changes scale granularity so the INT8 path can stay efficient on gfx942. MXFP4 keeps per-32 block scaling because gfx950 adds the scaled MFMA family needed to consume those scales directly.\n\n## The Kimi-K2.5-MXFP4 Model on gfx950[#](#the-kimi-k2-5-mxfp4-model-on-gfx950)\n\nMXFP4 combines two ideas:\n\n**4-bit element storage**: the value payload uses FP4 E2M1, packed two elements per byte.** Microscaling**: each block of 32 K-dimension values carries an E8M0 scale, giving the block a shared exponent.\n\nThe Kimi-K2.5-MXFP4 model configuration uses this scheme for both activations and weights:\n\nField |\nValue observed in |\n|---|---|\n|\n|\n|\n|\n|\n|\n|\n|\n|\n|\n\n### The Opportunity: Block-Scaled FP4 MFMA[#](#the-opportunity-block-scaled-fp4-mfma)\n\nThe W4A8 post focused on a gfx942 question: how do we move the MoE path from BF16 MFMA to an 8-bit MFMA path without losing too much accuracy? MXFP4 asks the next-generation gfx950 version of that question: if the hardware can consume block-scaled 4-bit floating-point operands directly, can we keep both the compact payload and a fine-grained scale layout in the matrix core?\n\nThat is the important architectural change. On CDNA3/gfx942, the common choices were BF16, INT8, FP8, and BF8 MFMA forms. On CDNA4/gfx950, the MFMA family adds block-scaled low-precision formats, including FP4, FP6, and BF6, with an E8M0 block exponent shared by 32 K-dimension elements [[7]](#references).\n\nPath |\nValue payload |\nScale placement |\nMatrix instruction implication |\n|---|---|---|---|\nBF16 |\n16 bits/value |\nNo external block scale |\nStable, simple, but higher data movement |\nFP8 / W8A8 |\n8 bits/value |\nEncoded in FP8 values and runtime scaling path |\nMature 8-bit floating-point path |\nW4A8 |\n4-bit weights, 8-bit activations |\nWeight and activation scales applied after INT8 accumulation |\nEfficient on gfx942, but per-group weight scales do not fit the INT8 inner loop |\nMXFP4 |\n4-bit FP4 payload, packed two values per byte |\nE8M0 scale per 32 K values |\ngfx950 scaled MFMA can consume payloads and block scales together |\n\nCompared with FP8, the main MXFP4 advantage is not just “a smaller number.” It is a different memory/computation trade-off:\n\n**Half the element bytes of FP8**: FP4 payloads are 4 bits instead of 8 bits, reducing weight and activation traffic for quantized matrix paths.** Hardware-visible block scales**: gfx950 can consume the FP4 payload and the E8M0 scale operands in scaled MFMA instructions, rather than requiring a full software dequantization back to BF16 first.**Better range than raw FP4**: the shared E8M0 exponent lets a 32-element block shift its dynamic range, which is essential for using FP4 in large MoE models.\n\nThere is still a trade-off. FP8 keeps more mantissa/exponent information per element and is often simpler for accuracy-sensitive paths. MXFP4 wins when the model can tolerate the lower per-element precision and when memory bandwidth, model footprint, or MoE weight traffic dominate.\n\n### MXFP4 Tensor Anatomy[#](#mxfp4-tensor-anatomy)\n\nMXFP4 is easiest to understand as a two-tensor representation for each quantized matrix:\n\n```\nFP4 payload:  [two E2M1 values packed into each uint8 byte]\nE8M0 scales: [one uint8 scale for every 32 original K-dimension values]\n```\n\nFor a representative Kimi-K2.5 expert `gate_proj`\n\nor `up_proj`\n\nshard, the logical weight matrix has 2,048 output channels and 7,168 K-dimension values per row. In the MXFP4 checkpoint, this appears as:\n\nTensor |\nStored dtype |\nStored shape |\nWhat it represents |\n|---|---|---|---|\n|\n|\n|\nTwo FP4 E2M1 values packed per byte, so the K dimension is halved in storage |\n|\n|\n|\nOne E8M0 scale for each block of 32 original K values ( |\n\nFor a representative expert `down_proj`\n\nshard, the same rule applies in the other projection shape:\n\nTensor |\nStored dtype |\nStored shape |\nWhat it represents |\n|---|---|---|---|\n|\n|\n|\nPacked FP4 payload for 2,048 logical K values |\n|\n|\n|\nOne E8M0 scale for each 32-value block ( |\n\nThe scale overhead is small compared with the payload. A 32-value block needs 16 bytes of FP4 payload plus 1 byte of E8M0 scale, or about 0.53125 bytes per original value before additional tensor metadata and alignment effects. The key point is that this scale is not a dequantization side table that must disappear before compute. On gfx950, it is part of the block-scaled operand stream consumed by the FP4 matrix instruction.\n\n```\nBF16 activations\n      |\n      v\ndynamic 1x32 MXFP4 quantization\n      |\n      +--> activation FP4 payload --+\n      +--> activation E8M0 scales --+\n                                    +--> gfx950 V_MFMA_SCALE_F32_* --> accumulator/output\n      +--> weight FP4 payload ------+\n      +--> weight E8M0 scales ------+\n```\n\n**Figure 1.** MXFP4 dataflow on gfx950. Both activations and weights are represented as packed FP4 payloads plus E8M0 block scales. The important difference from a storage-only FP4 path is that the block scales stay visible to the matrix instruction through the gfx950 scaled MFMA family.\n\n### Choosing the Right Weight Layout[#](#choosing-the-right-weight-layout)\n\nThe previous W4A8 post framed quantization as a layout problem as much as a datatype problem: the target MFMA instruction determines where scales can be applied, and that determines which checkpoint layout is efficient at runtime. MXFP4 follows the same logic, but with a different answer on gfx950.\n\nFormat |\nWeight payload |\nActivation path |\nScale layout |\nRuntime compute path |\nBest fit |\n|---|---|---|---|---|---|\n|\nINT4 |\nBF16 |\nPer-group, |\nDequantize to BF16, then BF16 MFMA |\nAccuracy-oriented INT4 serving when BF16 compute is acceptable |\n|\nINT4 |\nBF16 to INT8 dynamically |\nPer-channel weight scales plus per-token activation scales |\nINT8 MFMA with scale application in the epilogue |\nSmaller weight footprint with 8-bit compute on gfx942 |\n|\nFP8 |\nFP8 |\nFP8 weight and activation path |\nFP8 MFMA through AITER CK/ASM kernels |\nSimpler low-precision pipeline when HBM capacity allows |\n|\nFP4 E2M1 packed two values per byte |\nFP4 dynamic quantization for quantized matrix paths |\nE8M0 scale per 32 K-dimension values |\ngfx950 scaled MFMA with FP4 payloads and block scales |\nLowest payload width with hardware-visible block scaling |\n\nThe key MXFP4 difference is that the scale is not merely checkpoint metadata. The E8M0 scale is part of the block-scaled compute contract: ATOM maps the model metadata to 1x32 FP4 quantization, and AITER can dispatch gfx950 kernels that keep the FP4 payload and scale operands close to the matrix instruction.\n\n### When Is the Scale Applied? The gfx950 Scaled-MFMA Difference[#](#when-is-the-scale-applied-the-gfx950-scaled-mfma-difference)\n\nThe most important question is not only “how many bits does each value use?” It is “where does the scale enter the GEMM?”\n\nThat placement determines whether a scale layout is cheap, expensive, or structurally incompatible with the target MFMA instruction.\n\n**W4A16: scale is absorbed before MFMA.**\n\nIn the W4A16 path, INT4 weights are dequantized into BF16 before the matrix instruction:\n\n```\nw_bf16 = int4_to_bf16(w_int4) * group_scale[n, k // 32]\nacc += w_bf16 * act_bf16\n```\n\nAfter dequantization, the scale has been baked into the BF16 value. The BF16 MFMA instruction sees only BF16 operands; it does not know whether the original checkpoint had one scale per row or one scale per 32 values. This makes fine-grained per-group scaling accuracy-friendly, but it also keeps the compute path in BF16.\n\n**W4A8 on gfx942: no block-scaled MFMA, so scale is deferred until the epilogue.**\n\nIn the W4A8 INT8 path, the kernel wants to keep the inner loop entirely in the integer domain:\n\n```\nw_int8 = sign_extend(w_int4)          # no floating-point scale here\nint32_sum += w_int8 * act_int8        # INT8 MFMA, accumulates to INT32\noutput = weight_scale[n] * act_scale[token] * int32_sum\n```\n\nThe weight scale cannot be applied before INT8 MFMA, because multiplying by a floating-point scale would turn the operand into a floating-point value. The gfx942 W4A8 path also does not have a block-scaled MFMA instruction that can consume INT4 or INT8 payloads together with per-32 scale operands. The scale therefore moves to the epilogue, after the dot product is complete. That is why W4A8 prefers per-channel weight scales: one row scale can be applied once at the end. A per-group scale every 32 K values would force the dot product into many separately scaled partial sums, which defeats the clean INT8 inner loop.\n\n**MXFP4 on gfx950: scaled MFMA makes per-32 scaling a hardware compute path.**\n\nMXFP4 takes a different route. Its E8M0 scale is neither fully baked into a BF16 value before MFMA nor delayed as a single correction after accumulation. The scale is part of the block-scaled FP4 operand representation:\n\n```\nw_fp4x2, w_scale_e8m0 = load_mxfp4_weight_block(...)\na_fp4x2, a_scale_e8m0 = dynamic_quantize_activation_block(...)\nacc = scaled_mfma(w_fp4x2, a_fp4x2, w_scale_e8m0, a_scale_e8m0)\n```\n\nThis is the key gfx950 distinction. CDNA4 adds the `V_MFMA_SCALE_F32_*`\n\nfamily for block-scaled low-precision inputs, including FP4 [[7]](#references). The scaled MFMA path can consume low-precision FP4 payloads together with their block scales, so a 1x32 scale layout remains compatible with the high-throughput matrix instruction. In other words, MXFP4 keeps the local dynamic-range benefit of per-32 scaling without reverting the main compute path to BF16 dequantization or forcing all scale application into a late epilogue.\n\nFormat |\nScale applied |\nConsequence |\n|---|---|---|\n|\nBefore MFMA, during dequantization to BF16 |\nFine-grained per-group scales are natural, but compute uses BF16 MFMA |\n|\nAfter INT8 MFMA, in the epilogue |\nEfficient INT8 inner loop, but weight scales must be coarse enough to apply after accumulation |\n|\nEncoded in FP8 values and handled by FP8 compute path |\nSimple 8-bit floating-point pipeline, with a wider payload than FP4 |\n|\nAlongside FP4 payloads in gfx950 |\nPer-32 block scaling stays in the matrix path with 4-bit payloads |\n\nThis is why MXFP4 is more than “FP8 but smaller.” The format changes where scale arithmetic lives, and gfx950 is the hardware generation that turns that scale placement into a first-class MFMA path. On gfx950, the serving stack can keep FP4 values compact while still giving each 32-value block its own exponent-like scale at compute time.\n\n### Checkpoint Layout and Quantization Metadata[#](#checkpoint-layout-and-quantization-metadata)\n\nThe MXFP4 checkpoint carries the runtime contract in `config.json`\n\n, not only in tensor names. Both weights and activations use the same high-level quantization scheme:\n\nConfig field |\nWeight |\nInput activation |\n|---|---|---|\n|\n|\n|\n|\n|\n|\n|\n|\n|\n|\n|\n|\n|\n|\n|\n|\n|\n|\n\nThis split is important:\n\n**Weights are static**: the packed FP4 payload and E8M0`weight_scale`\n\ntensors are stored in the checkpoint.**Activations are dynamic**: ATOM quantizes live BF16 activation blocks into FP4 payloads and E8M0 scales at runtime.** Some layers remain outside MXFP4**: the checkpoint excludes attention projections, MoE router gates, the language model head, and vision/projector components from this global FP4 quantization path. That keeps precision-sensitive or non-MoE components on their own kernels while concentrating MXFP4 on the matrix-heavy MLP and expert projections.\n\nIn the checkpoint index, quantized modules appear as paired `weight`\n\nand `weight_scale`\n\ntensors. For Kimi-K2.5-MXFP4, the index contains 69,303 `weight_scale`\n\ntensors, matching 69,303 modules with stored block-scale metadata. That is the checkpoint-side reason ATOM can map the model to `QuantType.per_1x32`\n\nrather than treating the FP4 payload as a generic packed byte tensor.\n\n## Runtime Pipeline: From BF16 Activations to Scaled FP4 MFMA[#](#runtime-pipeline-from-bf16-activations-to-scaled-fp4-mfma)\n\nThe MXFP4 serving path matters because it keeps FP4 payloads close to the matrix core instead of expanding the model back to a wider format before the main compute. At runtime, the path has three connected stages: parse the checkpoint metadata, quantize live activations into the same 1x32 block-scaled format, then dispatch GEMM and MoE kernels that use gfx950 scaled MFMA.\n\n### Stage A: Quantization Config Parsing[#](#stage-a-quantization-config-parsing)\n\nATOM maps `per_group`\n\nquantization to `QuantType.per_1x32`\n\n. The same parser recognizes `fp4`\n\n, `float4`\n\n, and `mxfp4`\n\nas `fp4x2`\n\n, then forces MXFP4 into `per_1x32`\n\nblock scaling when needed.\n\nThis matches the model config: FP4 values with group size 32 and E8M0 scales.\n\n### Stage B: Dynamic Activation Quantization[#](#stage-b-dynamic-activation-quantization)\n\nFor weight tensors, the FP4 payload and E8M0 scales are already stored in the checkpoint. Activations are different: they are produced live in BF16 by the surrounding transformer layers, so ATOM dynamically quantizes activation blocks at inference time.\n\nFor the MXFP4 linear path, ATOM calls `get_hip_quant(QuantType.per_1x32)`\n\n, views activations as `torch.float4_e2m1fn_x2`\n\n, views scales as `torch.float8_e8m0fnu`\n\n, and passes both into the low-precision GEMM path. Conceptually:\n\n``` php\nhidden_bf16\n  -> dynamic 1x32 FP4 quantization\n  -> activation_fp4x2 + activation_e8m0_scale\n  -> scaled FP4 GEMM\n```\n\nThis is the activation-side counterpart to the checkpoint’s packed `weight`\n\nand `weight_scale`\n\ntensors.\n\n### Stage C: MXFP4 Linear GEMM[#](#stage-c-mxfp4-linear-gemm)\n\nDuring warmup, AITER loaded multiple gfx950 `f4gemm`\n\nkernels, including:\n\n```\nf4gemm_bf16_per1x32Fp4_BpreShuffle_256x256.co\nf4gemm_bf16_per1x32Fp4_BpreShuffle_192x128.co\nf4gemm_bf16_per1x32Fp4_BpreShuffle_128x128.co\n```\n\nThe loaded FP4 GEMM path uses the block-scaled MFMA instruction family:\n\n```\nv_mfma_scale_f32_16x16x128_f8f6f4 ...\n```\n\nThis is the key gfx950 instruction family for the FP4 path: the kernel is not simply converting FP4 weights to BF16 and using BF16 MFMA; it is using the scaled MFMA form that accepts low-precision block-scaled operands.\n\n### Stage D: MXFP4 Fused MoE[#](#stage-d-mxfp4-fused-moe)\n\nATOM’s `Mxfp4MoEMethod`\n\ncreates FP4 expert weights and uint8 scale tensors with a hard-coded MXFP4 block size of 32. For MoE execution, it passes:\n\n`quant_type=self.quant_type`\n\n`w1_scale=layer.w13_weight_scale`\n\n`w2_scale=layer.w2_weight_scale`\n\ninto AITER `fused_moe`\n\n.\n\nAITER selected FlyDSL 2-stage fused MoE kernels with FP4 dtypes:\n\n```\nflydsl_moe1_afp4_wfp4_bf16_t64x128x256_w4_bnt0_xcd4\nflydsl_moe2_afp4_wfp4_bf16_t64x256x256_reduce_xcd4\n```\n\nFor smaller token regimes, the run also used:\n\n```\nflydsl_moe1_afp4_wfp4_bf16_t32x64x256_w4\nflydsl_moe2_afp4_wfp4_bf16_t32x256x256_atomic_bnt2\n```\n\nThe dispatch line included:\n\n```\ntorch.float4_e2m1fn_x2, torch.float4_e2m1fn_x2, QuantType.per_1x32\n```\n\nTogether, these linear and MoE paths show the core benefit of MXFP4 on gfx950: FP4 payloads and 1x32 scaling can be carried into the compute kernels instead of serving only as a compressed checkpoint format.\n\n### Verifying Scaled MFMA in the Generated Kernels[#](#verifying-scaled-mfma-in-the-generated-kernels)\n\nThe scaled-MFMA claim was verified by disassembling representative gfx950 FP4 code objects used by the ATOM/AITER stack. The FP4 GEMM kernels contain `v_mfma_scale_f32_16x16x128_f8f6f4`\n\ninstructions, for example:\n\n```\nv_mfma_scale_f32_16x16x128_f8f6f4 a[0:3], v[136:139], v[8:11], a[0:3], v208, v200 ...\n```\n\nRepresentative `f4gemm_bf16_per1x32Fp4_*`\n\ncode objects contained hundreds of `v_mfma_scale_f32_16x16x128_f8f6f4`\n\ninstructions. The MXFP4 MoE code objects also contained the same scaled MFMA family, with typical counts in the hundreds per code object. This confirms that the serving path is not merely loading a compressed FP4 checkpoint and expanding it to BF16 before the matrix core; the gfx950 kernels really do execute the block-scaled MFMA path.\n\n### Non-MXFP4 Kernels Still Appear[#](#non-mxfp4-kernels-still-appear)\n\nThe serving stack also includes supporting kernels that are not MXFP4:\n\n`mla_a8w8_qh16_qseqlen1_gqaratio16_ps.co`\n\nfor MLA decodeBF16 GEMM fallback kernels such as\n\n`bf16gemm_fp32bf16_tn_80x64_splitk_clean.co`\n\nnormalization, quantization, cache, and sampling modules\n\nThis is expected. The model is not “FP4 everywhere.” The MXFP4 path is concentrated in quantized linear/MoE compute, while attention, routing, normalization, logits, and other framework components still use their own precision choices.\n\n## End-to-End Performance on 8x MI355X[#](#end-to-end-performance-on-8x-mi355x)\n\nWe benchmarked the ATOM OpenAI-compatible server using the same input/output shape used in the previous Kimi-K2.5 ROCm blog posts:\n\nTP = 8\n\nGPU = 8x MI355X / gfx950\n\nISL/OSL = 10,240 / 512\n\nlow concurrency = 2,\n\n`num_prompts = 10`\n\nhigh concurrency = 40,\n\n`num_prompts = 160`\n\n`request_rate = inf`\n\n`random_range_ratio = 1.0`\n\nThis keeps the benchmark shape aligned with the earlier MI300X and MI325X posts. The hardware and serving stack are different, so the numbers below should be read as the MI355X/gfx950 MXFP4 result under the same workload shape, not as a same-system speedup over W4A16, W4A8, or W8A8.\n\n### Test Environment[#](#test-environment)\n\nComponent |\nConfiguration |\n|---|---|\nHardware |\n8x AMD Instinct MI355X, gfx950 |\nServing stack |\nATOM OpenAI-compatible server with AITER kernels |\nModel |\n|\nTensor parallelism |\nTP = 8 |\nKV cache dtype |\n|\nDocker image |\n|\n\n### Launching the Server[#](#launching-the-server)\n\nThe following command shows the serving setup used for the 8-GPU run. Replace `/models/Kimi-K2.5-MXFP4`\n\nwith the location where the MXFP4 checkpoint is mounted inside the container.\n\n```\nexport IMAGE=\"rocm/atom-dev:latest@sha256:eba8c90854e5ce6a6f5b5ac4d35d41f7df76981229e5d91f32a9d2176c52a2e8\"\nexport MODEL_DIR=\"/models/Kimi-K2.5-MXFP4\"\nexport PORT=18002\n\ndocker run --rm --network=host \\\n    --device=/dev/kfd --device=/dev/dri \\\n    --group-add video \\\n    --cap-add=SYS_PTRACE \\\n    --security-opt seccomp=unconfined \\\n    --ipc=host --shm-size=256G \\\n    --ulimit memlock=-1 --ulimit stack=67108864 \\\n    -e HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \\\n    -v /path/to/Kimi-K2.5-MXFP4:${MODEL_DIR}:ro \\\n    ${IMAGE} \\\n    bash -lc \"cd /app/ATOM && \\\n      python3 -m atom.entrypoints.openai_server \\\n        --model ${MODEL_DIR} \\\n        --trust-remote-code \\\n        -tp 8 \\\n        --kv_cache_dtype fp8 \\\n        --host 0.0.0.0 \\\n        --server-port ${PORT}\"\n```\n\n### Benchmark Commands[#](#benchmark-commands)\n\nWe use the same random input/output lengths as the earlier Kimi-K2.5 posts: 10,240 input tokens and 512 output tokens.\n\n**Prefill-dominated (max concurrency = 2):**\n\n```\npython3 -m atom.benchmarks.benchmark_serving \\\n    --model=/models/Kimi-K2.5-MXFP4 \\\n    --backend=vllm \\\n    --base-url=http://127.0.0.1:18002 \\\n    --trust-remote-code \\\n    --dataset-name=random \\\n    --random-input-len=10240 \\\n    --random-output-len=512 \\\n    --random-range-ratio=1.0 \\\n    --num-prompts=10 \\\n    --max-concurrency=2 \\\n    --request-rate=inf \\\n    --percentile-metrics=ttft,tpot,itl,e2el\n```\n\n**Decode-dominated (max concurrency = 40):**\n\n```\npython3 -m atom.benchmarks.benchmark_serving \\\n    --model=/models/Kimi-K2.5-MXFP4 \\\n    --backend=vllm \\\n    --base-url=http://127.0.0.1:18002 \\\n    --trust-remote-code \\\n    --dataset-name=random \\\n    --random-input-len=10240 \\\n    --random-output-len=512 \\\n    --random-range-ratio=1.0 \\\n    --num-prompts=160 \\\n    --max-concurrency=40 \\\n    --request-rate=inf \\\n    --percentile-metrics=ttft,tpot,itl,e2el\n```\n\n**Concurrency = 2 (prefill-dominated):**\n\nMetric |\nStatistic |\nMXFP4 |\n|---|---|---|\n|\nMean |\n421.24 |\nMedian |\n380.50 |\n|\nP99 |\n607.20 |\n|\n|\nMean |\n11.30 |\nMedian |\n11.15 |\n|\nP99 |\n12.49 |\n|\n|\n- |\n165.30 |\n\n**Concurrency = 40 (decode-dominated):**\n\nMetric |\nStatistic |\nMXFP4 |\n|---|---|---|\n|\nMean |\n4,167.25 |\nMedian |\n3,992.10 |\n|\nP99 |\n8,897.46 |\n|\n|\nMean |\n29.34 |\nMedian |\n29.57 |\n|\nP99 |\n39.36 |\n|\n|\n- |\n1,055.95 |\n\nThe low-concurrency case highlights first-token latency and per-token decode cost when only two requests are active. The high-concurrency case stresses parallel decoding, where aggregate output throughput rises to 1,055.95 tok/s while mean TPOT increases to 29.34 ms/token.\n\nThese are end-to-end serving measurements, not isolated GEMM or MoE microbenchmarks. They include request scheduling, attention, MoE, sampling, KV cache management, and framework overhead around the MXFP4 kernels.\n\n### How to Interpret These Results[#](#how-to-interpret-these-results)\n\nThe strongest claim supported by this experiment is that Kimi-K2.5-MXFP4 can be served end-to-end on 8x MI355X with ATOM while exercising gfx950 scaled-MFMA FP4 kernels in the matrix-heavy paths. The benchmark includes the full serving stack, so it is more relevant to deployment than an isolated GEMM microbenchmark.\n\nThe comparison below is useful for context, but it is not a controlled same-system speedup experiment. The previous W4A16, W4A8, and W8A8 rows were measured on MI325X with SGLang/AITER, while the MXFP4 row was measured on MI355X with ATOM/AITER. That means several variables change at once: hardware generation, quantization format, serving framework, kernel dispatch path, and runtime scheduler behavior.\n\nFor this reason, the MXFP4 numbers should be read as a gfx950 enablement result and a method-aligned workload comparison. A strict speedup claim would require a same-system baseline, such as another Kimi-K2.5 format on the same 8x MI355X ATOM stack, or an MXFP4 fallback path on the same hardware.\n\nResults may vary based on system configuration, software versions, model settings, and workload characteristics.\n\n### Method-Aligned Comparison with the Previous Post[#](#method-aligned-comparison-with-the-previous-post)\n\nThe table below mirrors the benchmark comparison style from the W4A8/W8A8 post, but the interpretation is different. The W4A16, W4A8, and W8A8 rows were reported on an 8x MI325X SGLang/AITER setup; the MXFP4 row here was measured on an 8x MI355X ATOM/AITER setup. The workload shape and GPU count are aligned, but the hardware generation, quantization format, and serving stack are not identical, so this is context rather than a same-system speedup claim.\n\n**Concurrency = 2 (prefill-dominated):**\n\nFormat |\nSystem / stack |\nWeight layout |\nMean TTFT (ms) ↓ |\nMean TPOT (ms/token) ↓ |\nOutput throughput (tok/s) ↑ |\n|---|---|---|---|---|---|\nW4A16 |\n8x MI325X / SGLang |\nINT4 per-group, BF16 compute |\n723.96 |\n20.17 |\n92.78 |\nW4A8 |\n8x MI325X / SGLang |\nINT4 per-channel, INT8 compute |\n695.30 |\n16.95 |\n109.35 |\nW8A8 |\n8x MI325X / SGLang |\nFP8 weights and activations |\n747.52 |\n14.25 |\n127.47 |\nMXFP4 |\n8x MI355X / ATOM |\nFP4 E2M1 with E8M0 1x32 scales |\n421.24 |\n11.30 |\n165.30 |\n\n**Concurrency = 40 (decode-dominated):**\n\nFormat |\nSystem / stack |\nWeight layout |\nMean TTFT (ms) ↓ |\nMean TPOT (ms/token) ↓ |\nOutput throughput (tok/s) ↑ |\n|---|---|---|---|---|---|\nW4A16 |\n8x MI325X / SGLang |\nINT4 per-group, BF16 compute |\n10,183.74 |\n51.99 |\n556.94 |\nW4A8 |\n8x MI325X / SGLang |\nINT4 per-channel, INT8 compute |\n9,801.37 |\n46.78 |\n607.24 |\nW8A8 |\n8x MI325X / SGLang |\nFP8 weights and activations |\n9,764.04 |\n53.57 |\n551.12 |\nMXFP4 |\n8x MI355X / ATOM |\nFP4 E2M1 with E8M0 1x32 scales |\n4,167.25 |\n29.34 |\n1,055.95 |\n\nThe comparison highlights why MXFP4 is interesting on gfx950: it combines the small payload width of 4-bit weights with a scale layout that the hardware can consume directly in the FP4 matrix path. That is the key distinction from both W4A8, which changes scale granularity to make INT8 MFMA efficient, and W8A8, which uses a wider FP8 payload to keep the compute pipeline simple.\n\n### When Should You Choose MXFP4?[#](#when-should-you-choose-mxfp4)\n\nMXFP4 is attractive when the deployment target has gfx950 block-scaled MFMA support and the model is available in a checkpoint layout that preserves FP4 payloads plus E8M0 scales. In that setting, MXFP4 gives the serving stack a compact 4-bit payload without forcing per-32 scales to be baked into BF16 before compute or delayed into a single epilogue scale.\n\nFormat |\nBest fit |\nMain trade-off |\n|---|---|---|\n|\ngfx942-era deployments where INT8 MFMA is the optimized path and INT4 weight footprint matters |\nPer-channel weight scales are needed for the INT8 inner loop, which is coarser than per-group scaling |\n|\nDeployments that prefer a simpler FP8 pipeline and can afford the larger weight footprint |\nFP8 payloads are wider than FP4, so memory traffic and model footprint are higher |\n|\ngfx950 deployments that can use block-scaled FP4 MFMA and want the smallest payload width with per-32 block scaling |\nRequires gfx950 scaled-MFMA support and an MXFP4-aware serving stack |\n\nIn practice, MXFP4 is a good candidate when weight bandwidth, MoE expert traffic, or model footprint are important constraints, and when the workload can tolerate FP4 element precision with E8M0 block scaling. FP8 remains a simpler choice when maximum precision margin or broader kernel availability is more important than the smallest payload. W4A8 remains useful for gfx942 paths where INT8 MFMA is the available high-throughput option.\n\n## Accuracy Validation: GSM8K[#](#accuracy-validation-gsm8k)\n\nTo validate that the ATOM MXFP4 path preserves model quality on a downstream reasoning benchmark, we ran GSM8K 10-shot evaluation against the same ATOM server. This matches the GSM8K protocol used in the previous W4A8/W8A8 post. The run used greedy generation (`do_sample=False`\n\n, `temperature=0.0`\n\n) and evaluated the full GSM8K test set of 1,319 samples.\n\nThe evaluation used `lm-eval`\n\nthrough the server’s OpenAI-compatible completions endpoint:\n\n```\nlm_eval \\\n    --model local-completions \\\n    --model_args model=/models/Kimi-K2.5-MXFP4,base_url=http://127.0.0.1:18002/v1/completions,tokenized_requests=False,tokenizer_backend=None,num_concurrent=32 \\\n    --tasks gsm8k \\\n    --num_fewshot 10 \\\n    --batch_size 1\n```\n\nThe `strict-match`\n\nfilter follows the canonical GSM8K answer marker, while the `flexible-extract`\n\nfilter extracts the final numeric answer more permissively. Reporting both filters makes the result easier to compare with the earlier Kimi-K2.5 posts.\n\nBenchmark |\nFilter |\nn-shot |\nMetric |\nValue |\nStderr |\n|---|---|---|---|---|---|\nGSM8K |\nflexible-extract |\n10 |\nexact_match |\n0.9401 |\n0.0065 |\nGSM8K |\nstrict-match |\n10 |\nexact_match |\n0.9401 |\n0.0065 |\n\nThe result shows that the MXFP4 serving path maintains strong downstream reasoning quality while using the lower-precision FP4 payload and gfx950 block-scaled compute path.\n\n## Summary[#](#summary)\n\nThis third Kimi-K2.5 optimization note extends the earlier MI300X/MI325X story to MI355X and gfx950 by moving from kernel and quantization work to optimized serving:\n\n**ATOM makes the optimized path deployable**: users can run a standalone OpenAI-compatible server, while the vLLM OOT plugin path lets vLLM remain the serving framework and ATOM provide AMD-optimized model execution.**The checkpoint is truly MXFP4**: the model configuration records FP4 tensors with`group_size=32`\n\nand E8M0 scales, and stores packed FP4 payloads plus`weight_scale`\n\ntensors for quantized modules.**ATOM parses MXFP4 into a 1x32 FP4 execution path**:`per_group`\n\nmaps to`QuantType.per_1x32`\n\n, and MXFP4 is represented as`torch.float4_e2m1fn_x2`\n\nplus`torch.float8_e8m0fnu`\n\nscales at runtime.**AITER dispatches gfx950 FP4 kernels**: the serving path uses`f4gemm`\n\nkernels and FlyDSL FP4 fused MoE kernels built around gfx950 scaled MFMA support.**End-to-end serving is strong on 8x MI355X**: under the aligned 10,240/512 workload, MXFP4 reached 165.30 output tok/s at concurrency 2 and 1,055.95 output tok/s at concurrency 40.**Accuracy is maintained on GSM8K**: the same ATOM server scored 94.01% flexible-extract exact match on GSM8K 10-shot (`0.9401 +/- 0.0065`\n\n) on the full 1,319-sample test set.\n\nThe broader takeaway is that gfx950 turns MXFP4 into a first-class compute format, and ATOM turns that hardware capability into a serving path. Compared with FP8, MXFP4 reduces data movement through 4-bit payloads while preserving dynamic range through block scales; compared with older FP4-as-storage approaches, gfx950 scaled MFMA lets the kernel keep the low-precision representation much closer to the matrix core. ATOM is the framework layer that makes this usable: it loads the HuggingFace checkpoint, maps quantization metadata to runtime operators, chooses the AITER/FlyDSL kernels, exposes a serving API, and provides benchmark tooling for the full stack rather than isolated kernels.\n\n## References[#](#references)\n\n[2] [Kimi-K2.5-MXFP4 on HuggingFace](https://huggingface.co/amd/Kimi-K2.5-MXFP4)\n\n[3] [ATOM](https://github.com/ROCm/ATOM)\n\n[4] [AITER](https://github.com/ROCm/aiter)\n\n[5] [vLLM](https://github.com/vllm-project/vllm)\n\n[6] [AMD Quark](https://github.com/amd/quark)\n\n[7] [AMD Instinct CDNA4 Instruction Set Architecture Reference Guide](https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/instruction-set-architectures/amd-instinct-cdna4-instruction-set-architecture.pdf)\n\n[8] [Previous ROCm blog: Accelerating Kimi-K2.5 on AMD Instinct MI300X: Optimizing Fused MoE with FlyDSL](https://rocm.blogs.amd.com/artificial-intelligence/kimi-k2.5-optimize/README.html)\n\n## Disclaimers[#](#disclaimers)\n\nThe information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities that cannot be completely prevented or mitigated. AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED ‘AS IS.” AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. AMD, the AMD Arrow logo, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies. © 2026 Advanced Micro Devices, Inc. All rights reserved", "url": "https://wpnews.pro/news/serve-kimi-k2-5-mxfp4-on-mi355x-with-atom", "canonical_source": "https://rocm.blogs.amd.com/software-tools-optimization/kimi-k25-mxfp4-atom/README.html", "published_at": "2026-07-23 00:00:00+00:00", "updated_at": "2026-07-23 16:56:42.276181+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-infrastructure", "ai-tools", "ai-products"], "entities": ["AMD", "Kimi-K2.5-MXFP4", "AMD Instinct MI355X", "ATOM", "AITER", "vLLM", "HuggingFace", "ROCm"], "alternates": {"html": "https://wpnews.pro/news/serve-kimi-k2-5-mxfp4-on-mi355x-with-atom", "markdown": "https://wpnews.pro/news/serve-kimi-k2-5-mxfp4-on-mi355x-with-atom.md", "text": "https://wpnews.pro/news/serve-kimi-k2-5-mxfp4-on-mi355x-with-atom.txt", "jsonld": "https://wpnews.pro/news/serve-kimi-k2-5-mxfp4-on-mi355x-with-atom.jsonld"}}