Scaling MiniMax-M3 Inference with Distributed Serving and Operator Co-Design on AMD Instinct MI355X GPUs AMD has optimized MiniMax-M3 inference on Instinct MI355X GPUs using ATOM, AITER, and ATOMesh, achieving improvements in serving throughput, token latency, and accuracy through online quantization, sparse attention, FP8 KV cache, EAGLE3 speculative decoding, and disaggregated serving. Scaling MiniMax-M3 Inference with Distributed Serving and Operator Co-Design on AMD Instinct MI355X GPUs scaling-minimax-m3-inference-with-distributed-serving-and-operator-co-design-on-amd-instinct-mi355x-gpus This blog walks you through a set of MiniMax-M3 inference optimizations on AMD Instinct™ MI355X GPUs using ATOM https://github.com/ROCm/ATOM , AITER https://github.com/ROCm/aiter , and ATOMesh https://rocm.blogs.amd.com/software-tools-optimization/atomesh-inference/README.html . For broader background on the inference engine and distributed serving layers used here, see the ROCm blogs on ATOM https://rocm.blogs.amd.com/software-tools-optimization/atom-inference-engine/README.html and ATOMesh https://rocm.blogs.amd.com/software-tools-optimization/atomesh-inference/README.html . We’ll break down: System-level bottlenecks across quantization, attention, KV cache, speculative decoding, and disaggregated inference. ATOM online quantization for faster attention GEMMs. AITER sparse attention, FP8 KV cache, and page-16 SHUFFLE layout. EAGLE3 speculative decoding for reducing target-model forward work. ATOMesh P/D disaggregation for distributed MiniMax-M3 serving. By the end, you’ll understand how these optimizations work together to improve serving throughput, reduce token latency, and maintain accuracy on MiniMax-M3 workloads. Why MiniMax-M3 Optimization Matters why-minimax-m3-optimization-matters MiniMax-M3 is a high-performance large language model with a complex inference path. It combines Mixture-of-Experts MoE , sparse attention, long-context KV cache management, and quantized checkpoints. Optimizing only one kernel is not enough. To unlock higher serving efficiency on AMD Instinct™ MI355X GPUs, the full inference path must be considered. The recent MiniMax-M3 optimization work focuses on four major areas: Area | Optimization | |---|---| Quantization | Online quantization for attention linear layers and GEMM acceleration | Attention runtime | AITER sparse attention, FP8 KV cache, and page-16 SHUFFLE KV layout | Speculative decoding | EAGLE3 speculative decoding with optimized draft and target execution | Disaggregated inference | ATOMesh P/D disaggregation with MiniMax-M3 sparse indexer-key cache transfer | Together, these changes improve the most important serving bottlenecks: attention GEMM cost, KV cache bandwidth, sparse attention scheduling, and target model forward frequency. Online Quantization for Attention GEMMs online-quantization-for-attention-gemms ATOM provides several online quantization methods that prepare checkpoint weights for efficient GEMM execution without regenerating the full model. These include quantizing BF16 weights to FP8 or FP4, and converting between formats within the same data type, such as FP8 block-scale weights to FP8 PTPC weights. This is useful for MiniMax-M3 quantized checkpoints because performance-sensitive paths, especially attention, can use lower-bit GEMMs or keep FP8 while switching to a different quantization scheme like PTPC. For the full set of supported online quantization modes and configuration options, see the ATOM online quantization guide https://github.com/ROCm/ATOM/blob/main/docs/online quantization guide.md . For MiniMax-M3, ATOM applies online quantization only where it improves GEMM throughput: attention linear layers and selected dense MLP layers. Modules such as lm head , embeddings, vision components, multimodal projectors, and MoE layers are excluded, so MoE weights keep their existing quantized format. This selective policy covers both MiniMaxAI/MiniMax-M3-MXFP8 and Quark quantized models such as amd/MiniMax-M3-MXFP4 , where some attention layers remain BF16 and can still benefit from PTPC FP8 online quantization. For implementation details, see ROCm/ATOM 1335 https://github.com/ROCm/ATOM/pull/1335 and ROCm/ATOM 1370 https://github.com/ROCm/ATOM/pull/1370 . Taking MiniMaxAI/MiniMax-M3-MXFP8 as an example, selected attention GEMM weights are loaded from their 1x32 block-scale checkpoint representation and converted to PTPC FP8 during model loading. The conversion uses GPU elementwise kernels, so it adds only a small one-time weight-loading overhead. After the PTPC weights are materialized, request-time inference uses them directly and does not pay additional online quantization latency. Figure 1 shows this load-time conversion path and the modules that remain in their original format. Figure 1. MiniMax-M3 online quantization path. An example online quantization configuration is shown below: --online quant config '{"global quant config": "ptpc fp8", "exclude layer": "lm head", "model.embed tokens", "vision tower", "multi modal projector", "patch merge mlp", " block sparse moe" }' AITER Sparse Attention and FP8 KV Cache aiter-sparse-attention-and-fp8-kv-cache The second optimization area targets attention runtime. This is not a single-point kernel change; it requires joint design and tuning between ATOM as the inference framework and AITER as the operator library. AITER provides high-performance attention operators for prefill and decode, while ATOM exposes the sparse metadata, KV-cache layout, and page-size configuration needed to use those operators efficiently. MiniMax-M3 also has model-specific sparse attention behavior, including grouped query heads and per-KV-head block selection. Those details determine how the sparse indexer, block table, and page-16 KV cache layout must be represented. Specifically, MiniMax-M3 has q head=64 and kv head=4 , which can be viewed as 4 KV-head sparse attention groups, each with q head=16 and kv head=1 . For each query token and KV head, the indexer selects 16 logical blocks, and each logical block covers 128 tokens. Different KV heads have separate top-k indexers and can select different KV blocks. Figure 2 shows how those logical blocks map to the page-16 SHUFFLE KV cache layout used by the AITER paged-attention path. Figure 2. MiniMax-M3 KV cache layout. The concrete implementation of the layout shown above spans both ATOM and AITER: On the ATOM side, we refactored MiniMax-M3 sparse attention into the standard Attention and PagedAttentionImpl framework, introduced SparseMHAPagedAttentionImpl , and added a page-16 SHUFFLE KV cache path with AITER gluon paged-attention runners for decode and prefill. This lets MiniMax-M3 reuse standard attention metadata binding while preserving model-specific sparse behavior, and allows the sparse index top-k path to directly emit compacted sparse block tables, avoiding an additional block-table build launch and reducing HBM round trips. For implementation details, see ROCm/ATOM 1334 https://github.com/ROCm/ATOM/pull/1334 .On the AITER side, we added SHUFFLE KV cache layout support to fused qknorm idxrqknorm , with separate K and V cache tensors and an asm layout flag for selecting the page-16 SHUFFLE addressing path. We also improved FP8 correctness by replacing hardcoded FP8 maximum values with AITER’s opus::finfo