{"slug": "day-0-support-for-mistral-large-4-on-vllm", "title": "Day 0 support for Mistral Large 4 on vLLM", "summary": "VLLM shipped day-0 support for Mistral Large 4, Mistral's new frontier mixture-of-experts native multimodal model, in the vllm/vllm-openai:mistral-large-4 Docker image, which also includes mistral-common 1.13. The 1-trillion-parameter model activates 49 billion parameters per token (52 billion including embeddings and output layers) and runs on vLLM's model runner V2 with breakable CUDA graphs, with FP8 and NVFP4 checkpoints both defaulting to an FP8 latent KV cache. vLLM reports the built-in DSpark drafter accepts about 2.9 tokens per step for roughly 2x output throughput on TP8, and that NVFP4 stays within 2.1 points of FP8 on every evaluated task with a mean difference of −0.3 points while decoding 1.08–1.19x faster per token.", "body_md": "# Day 0 support for Mistral Large 4 on vLLM\n\nWe’re excited to announce day-0 support for **Mistral Large 4** in vLLM. Mistral Large 4 is\nMistral’s new frontier mixture-of-experts native multimodal model. It combines a DeepSeek-style sparse attention\nstack (MLA plus a DSA top-k indexer) with several new architecture features: cross-layer KV\nsharing, NoPE layers, SSMax and LeRoPE. It ships with a built-in DSpark speculative decoding\ndrafter and a Pixtral vision encoder.\n\nSupporting it took more than a model file. ML4 runs on vLLM’s model runner V2 (MRV2) with breakable\nCUDA graphs. Its building blocks live in a new `vllm/models/mistral/` package, and the work produced\ngeneric improvements to decode context parallelism (DCP), prefill context parallelism (PCP),\nDSpark, NIXL-based disaggregated serving and HiSparse host-memory KV offload.\n\n## Highlights\n\n- **Architecture:** 72-layer MoE with 128 routed experts (top-4) plus 1 shared expert and a\nhidden size of 6144. Attention is MLA (KV LoRA rank 512, RoPE dim 64) with a DSA sparse indexer\nthat selects the top 2048 keys per query. 1 trillion parameters in total, 49 billion active per\ntoken (52 billion including embeddings and output layers).\n- **Checkpoints:** FP8 (block-scaled weights) and NVFP4 (NVFP4 experts, FP8 attention/shared\nexperts). Both use an FP8 latent KV cache (`fp8_ds_mla` ) by default.\n- **Multimodal:** a Pixtral vision encoder for image inputs, including multiple images per prompt.\n- **Speculative decoding:** a built-in DSpark drafter (3 sliding-window MLA layers, depth 3)\naccepts ~2.9 tokens per step and gives about 2x output throughput on TP8.\n- **Long context:** DCP gives ~7.9x more KV capacity on TP8. PCP+DCP prefill cuts time to first token (TTFT) by up to 1.6x. DSpark works under both.\n- **Disaggregated serving:** NIXL prefill/decode (P/D) disaggregation with DP/EP, TP and PCP×DCP prefill\nnodes, with DSpark on both sides.\n- **HiSparse:** host-memory sparse KV offload, now with support for cross-layer KV sharing.\n- **AMD:** day-0 on AMD Instinct MI350 Series GPUs with ROCm, including DSpark, image inputs and expert parallelism with MXFP4 experts.\n\n## Usage\n\nMistral Large 4 support ships in the `vllm/vllm-openai:mistral-large-4` Docker image, which also\nincludes `mistral-common` 1.13. The commands below run `vllm serve` inside it:\n\n```\ndocker pull vllm/vllm-openai:mistral-large-4\n```\n\n**FP8, one node with 8 GPUs, or two GB200/GB300 nodes with 4 GPUs each.** The weights need about\n1.1 TB of GPU memory.\n\n```\nvllm serve mistralai/Mistral-Large-4-1T-A52B \\\n  --tensor-parallel-size 8 \\\n  --speculative-config '{\"method\": \"dspark\", \"num_speculative_tokens\": 3}' \\\n  --tool-call-parser mistral --enable-auto-tool-choice \\\n  --reasoning-parser mistral\n```\n\nOn two 4-GPU nodes, add `--nnodes 2 --node-rank <0|1> --master-addr <node0> --distributed-executor-backend mp`,\nand `--headless` on node 1.\n\nThe single-node 8-GPU setup is validated on B200: TP8 with DSpark scores within noise of the disaggregated runs on GSM8K, and NVFP4 1P1D on 8+8 B200 with P DP8/EP → D TP8 reaches 164 output tok/s/GPU at 32k input (100 tok/s/user) and 83 at 60k (150 tok/s/user). B200 per-GPU throughput is about 0.8x GB300 in the same layouts.\n\n**NVFP4.** The NVFP4 checkpoint uses about 76 GiB of weights per GPU, compared with about 128 GiB\nfor FP8, so more memory is left for KV cache.\n\n```\nvllm serve mistralai/Mistral-Large-4-1T-A52B-NVFP4 \\\n  --tensor-parallel-size 8 \\\n  --speculative-config '{\"method\": \"dspark\", \"num_speculative_tokens\": 3}'\n```\n\nNVFP4 does not trade quality for speed. On multimodal, long-context, agentic coding, business automation and cyber evals it stays within 2.1 points of FP8 on every task, and the mean difference is −0.3 points. Token consumption is matched too: mean completion tokens stay within about 4% of FP8. It is also faster and lighter: at TP8, NVFP4 decodes 1.08–1.19x faster per token than FP8 without spec decoding (1.18–1.22x with DSpark) and holds ~1.5x the KV cache.\n\n*Comparison of FP8 versus NVFP4 scores and token consumption on core evaluations. Error bars show 95% confidence intervals estimated by bootstrapping. Across all domains, including multimodal reasoning (GDP.pdf, MathVision), long-context (Graphwalks 512k), coding agents (DeepSWE, SWE-bench Pro, Terminal-Bench 4), business automation (AutomationBench) and cyber (CyberGym), the NVFP4 checkpoint matches FP8 behaviour.\n\n**Long context (DCP).** Add `--decode-context-parallel-size 8` to shard the latent KV cache across\nthe 8 ranks. It requires the default FP8 KV cache.\n\n**Multimodal.** Pass `--limit-mm-per-prompt '{\"image\": N}'` to allow up to N images per prompt.\n\nFor tested deployment configurations (single-node, multi-node, disaggregated P/D and hardware-specific settings), see the\n[Mistral Large 4 recipe](https://recipes.vllm.ai/mistralai/Mistral-Large-4-1T-A52B).\n\n## Implementation details\n\n### Model architecture\n\n| component | Mistral Large 4 | \n|---|---|\n| layers / hidden size / heads | 72 / 6144 / 64 | \n| attention | MLA, no query LoRA, KV LoRA rank 512, RoPE dim 64 | \n| sparse attention | DSA indexer, top-k 2048, 32 index heads; 44 layers own an indexer, 28 reuse the top-k of the nearest earlier indexer layer | \n| positional encoding | YaRN + LeRoPE (learned per-frequency scaling); every 4th layer is NoPE | \n| query scaling | SSMax (length-dependent query scale) on every attention layer and indexer | \n| KV sharing | 9 layers (39, 43, …, 71) read layer 35’s latent cache (1 every 4) | \n| MoE | layer 0 dense; layers 1–71: 128 experts, top-4, 1 shared expert | \n| output | logit soft cap 30 | \n| vision | Pixtral encoder (30 layers, hidden 2048, patch 14) | \n\n*The 72 decoder layers: every 4th layer is NoPE, 28 layers reuse the top-k of the closest preceding indexer layer, the NoPE layers 39–71 read layer 35’s latent KV, and only layer 0 is dense.*\n\nWhat it took to support each piece in vLLM:\n\n**Sparse MLA + DSA indexer.** ML4 reuses vLLM’s sparse-MLA stack (FlashMLA sparse with the\n`fp8_ds_mla` latent cache, plus the sparse indexer with an FP8 indexer cache). ML4-specific\ndifferences are layered on top: shared-indexer layers that reuse the top-k of the nearest earlier indexer layer, an\nindexer with its own SSMax, and NoPE indexers on NoPE layers. Per token, the KV cache stores\n47,136 bytes: 41,328 for the MLA latent and 5,808 for the indexer.\n\n**Cross-layer KV sharing.** Nine NoPE layers read the latent KV written by layer 35 instead of\nwriting their own. We added `kv_sharing_target_layer_name` to `MLAAttention` and\n`AttentionLayerBase`. Consumer layers skip the cache write, and the V2 runner resolves them to the\ntarget layer’s KV cache spec. Consumers take no KV memory: 63 of the 72 layers own a cache.\nThis is model-agnostic and is being upstreamed in\n[vllm#64095](https://github.com/vllm-project/vllm/pull/64095).\n\n**NoPE, SSMax and LeRoPE.** NoPE layers are expressed as MLA with zero RoPE dims, so they use the\nsame kernels and cache layout. LeRoPE is folded into the YaRN frequencies. SSMax is fused into the\nMLA RoPE kernel (see fusions below).\n\n**A Mistral model package.** Rather than adding flags to `deepseek_v2.py`, the port builds on\nreusable Mistral components under `vllm/models/mistral/`: attention, indexer, SSMax, MLP, MoE,\nfused ops and a per-layer spec (`common/layer_spec.py`). The ML4 model itself is\n`vllm/models/mistral_large4/nvidia/model.py`, and the DSpark drafter sits next to it\n(`dspark.py`). Per-layer features (RoPE/NoPE, indexer owner/sharer, KV owner/consumer) come from\na layer spec built from `params.json`, not from model-specific branches in shared code.\n\n**Model runner V2.** ML4 runs only on MRV2. The V2 attention utilities discover KV-sharing layers\nand look up their target specs. DSpark plugs into the V2 DFlash/DSpark speculator.\n\n**Breakable CUDA graphs and fusions.** ML4 runs without `torch.compile`. It uses breakable CUDA\ngraphs, enabled automatically for the architecture (also when the text model sits inside the\nmultimodal wrapper). They gave about a 6x speedup over eager mode.\nHand-written fusions replace what the compiler would otherwise do, while enabling further optimizations:\n\n- First round: SSMax, MLA RoPE + SSMax, the indexer Q path and the indexer K path. These cut decode TPOT by 8–10%.\n- Decode round: FP32 router GEMM, a one-launch Triton LayerNorm for the indexer query, all-reduce + residual + RMSNorm in one FlashInfer kernel, and indexer K written straight into the FP8 cache. On TP8 FP8, c1 TPOT dropped from 10.22 to 9.00 ms (-12%) and c8 from 16.01 to 15.08 ms.\n\n### Long context: decode and prefill context parallelism\n\n**DCP.** MLA keeps one latent KV head, so under TP the cache is replicated on every rank. DCP shards it\nby token instead. On TP8, DCP8 raises KV capacity from 2.38M to 18.88M tokens (FP8) and from 3.60M to 28.76M\ntokens (NVFP4).\n\n*Concurrent 128k / 256k / 1M-token requests whose KV cache fits on 8× GB300 with ML4 NVFP4, TP8 vs TP8 DCP8: floor(KV pool / input length) - 3.60M and 28.76M tokens, gpu-memory-utilization 0.85.*\n\nOut of the box, DCP with sparse MLA paid a large fixed cost per decode step. A profile at 60k context showed about 7 ms per step of DCP-only work:\n\n- NCCL all-gathers of the LSE and of the indexer top-k candidates (44 indexer layers);\n- reduce-scatters;\n- a stable top-k merge;\n- query gathers;\n- small mask kernels and launch gaps.\n\nWe replaced these with:\n\n- **a2a combine** for FlashMLA sparse (merged as[vllm#61285](https://github.com/vllm-project/vllm/pull/61285) ):\na symmetric-memory all-to-all LSE combine instead of all-gather + reduce-scatter. ML4 uses it\nby default.\n- **Direct top-k exchange:** each rank pushes its packed (score, id) candidates into its peers’\nsymmetric-memory inboxes, and a selector reads them in place. This replaces an NCCL all-gather\nand a 6-pass radix select per indexer layer.\n- **Top-k compaction:** for batches of 32+ tokens, each rank compacts its valid top-k slots and\nbounds the kernel with`topk_length` , instead of scanning 2048 slots full of`-1` . In a 5k-token\nprefill profile, attention went from 165 ms to 38 ms.\n- A fused kernel neutralizes empty rows (a rank that owns none of a query’s top-k slots).\n\nThe DCP-specific overhead fell to 3.2 ms per step. Together with the decode fusions, on TP8 DCP8 FP8 (60k input / 1k output):\n\n|  | before | after | \n|---|---|---|\n| c1 TTFT p50 | 6.89 s | 4.73 s | \n| c1 TPOT p50 | 17.82 ms | 12.50 ms | \n| c64 output tok/s | 139.1 | 204.0 (+47%) | \n\n*TTFT and TPOT p50 vs concurrency on TP8 DCP8 (60k input / 1k output, no spec), before (a2db3f0b9d) and after (bbe5926511) the DCP collectives port and decode fusions.*\n\nThese DCP optimizations ship with Mistral Large 4 support on day 0.\n\n**When to use DCP.** DCP trades per-step latency for KV capacity. On the same 8x GB300 NVFP4\ndeployment and workload (no spec decoding), DCP8 decodes about 16–28% slower per token than TP8:\nc1 TPOT is 11.1 vs 8.7 ms at 8k input and 11.4 vs 9.1 ms at 60k. With up to 128 concurrent\nrequests of 8k–60k tokens, TP8’s 3.6M-token pool is never the bottleneck, so TP8 delivers more\nthroughput there. DCP8 pays off once KV is the limit: very long contexts, or many concurrent\nlong-lived sessions. In our agentic P/D runs (~100k-token sessions), a TP4 DCP4 decode node kept\nTTFT at 0.7 s at 32 sessions, where a plain TP4 decode node ran out of KV and climbed to 15 s.\n\n**PCP + DCP prefill.** For long prompts, PCP splits one prompt’s tokens across ranks. PCP+DCP\nnow works on ML4 with CUDA graphs, including DCP > PCP and DSpark. Two fixes made this work:\n\n- Under PCP the MLA KV-cache write first all-gathers its inputs across PCP ranks, and that gather depends on the batch’s prefill/decode split. Captured in a piecewise graph, the split was frozen at capture time. Now only the gather runs as an eager break, into fixed-size buffers, and the cache write itself stays in the graph.\n- The DCP group’s ranks are ordered to match the order collectives return data in.\n\nWith graphs on TP1×PCP4×DCP4, GSM8K-200 went from 0.315 to 0.975. With DSpark it reaches 0.980. As a P/D prefill node, PCP4×DCP4 gives the lowest TTFT we measured (see the P/D section).\n\n### DSpark speculative decoding\n\nML4 ships a DSpark drafter inside the checkpoint (`mtp_dspark.*` tensors and an `mtp` block in\n`params.json`). DSpark is a parallel drafter: a small non-causal transformer proposes all draft\npositions of a block in one forward pass. It conditions on hidden states tapped from target\nlayers 1, 18, 36, 54 and 71. ML4’s drafter adds:\n\n- **an RNN Markov head:** a gated recurrent state (rank 512) runs across the draft positions, so\neach position’s logits depend on the tokens drafted before it;\n- **position-specific mask inputs:** each draft slot gets its own learned input embedding instead\nof one shared mask token;\n- **sliding-window MLA draft layers:** 3 MLA layers with a 2048-token window and a BF16 KV cache,\nin their own KV cache group. The Triton MLA backend gained sliding-window decode for them.\n\n*One DSpark step: target taps feed the draft’s context K/V, the [anchor, mask_1, mask_2] block runs through 3 sliding-window MLA layers in one pass, the RNN head emits 3 draft tokens in sequence, and the target verifies them.*\n\nEnable it with `--speculative-config '{\"method\": \"dspark\", \"num_speculative_tokens\": 3}'`.\n\n**Results.** TP8, 200 real prompts, greedy (temperature 0), up to 512 output tokens:\n\n|  | FP8 base TPOT | FP8 DSpark TPOT (AL) | NVFP4 base TPOT | NVFP4 DSpark TPOT (AL) | \n|---|---|---|---|---|\n| c1 | 8.84 ms | 3.50 ms (2.90) | 8.21 ms | 2.95 ms (2.93) | \n| c8 | 15.28 ms | 7.08 ms (2.85) | 12.85 ms | 5.80 ms (2.86) | \n| c32 | 29.92 ms | 11.57 ms (2.85) | 25.38 ms | 9.80 ms (2.86) | \n\nThe table uses greedy decoding. With sampling, acceptance stays close: at temperature 1.0 /\ntop-p 0.95, AL is ~2.8 at concurrency 8. At higher temperatures, set\n`\"draft_sample_method\": \"probabilistic\"` in the speculative config: at T=1.5 it raises AL from\n2.04 to 2.36–2.53 (+12–22% throughput). Both modes are lossless thanks to standard rejection\nsampling.\n\nAt the same concurrency, DSpark raises output throughput 1.9–2.1x on FP8 and 1.9–2.2x on NVFP4. It doesn’t change what the model generates: every drafted token is verified against the target model, and we checked this empirically. When the base model re-scores DSpark’s outputs, they match its own greedy choices as often as the base model’s own outputs do. On GSM8K (1,319 questions), FP8 scores 0.909 without DSpark and 0.918 with it.\n\n*Output tok/s per GPU vs tok/s per user (1000 / TPOT p50), without (dashed) and with DSpark (solid, depth 3). Short: the 200-prompt TP8 runs in the table above. 8k / 32k / 60k: ML4 NVFP4 colocated on 8× GB300, TP8 and TP8 DCP8, `vllm bench serve` with real-text prompts and 1k output tokens; at 60k only the DSpark runs exist.*\n\n**DSpark + DCP.** Before this work, sliding-window MLA drafts could not run under DCP. Now the\ndraft’s small window cache (3 layers × ~2k tokens, about 7 MB per request) is replicated on every\nDCP rank, while the target cache stays DCP-sharded. This extends the existing\nsupport (vllm#56723) to windowed MLA drafts. On TP8 DCP8 at 60k context, c1 TPOT goes from 11.82 to \n5.22 ms (2.26x, AL 2.85), and GSM8K-200 stays at 0.965.\nThe cost is KV capacity: under DCP8 each 64-token draft block holds a block ID whose target\npages cover 512 tokens. At 64k context, capacity falls to about 0.69x of plain DCP8. That limit\ndoesn’t bind at `max_num_seqs` 128 below ~99k average context.\n\n**DSpark in P/D.** A DCP prefill node can hand off to a TP decode node with DSpark on both sides.\nNIXL pulls the replicated draft window from one prefill rank and the target KV in DCP slices.\nAcceptance is the same with any P/D layout: ~2.98 / 2.86 / 2.78 at 8k / 32k / 60k input.\n\nSliding-window MLA draft layers and DSpark under DCP are being upstreamed in\n[vllm#63795](https://github.com/vllm-project/vllm/pull/63795) and\n[vllm#63796](https://github.com/vllm-project/vllm/pull/63796).\n\n### Disaggregated serving with NIXL\n\nWe benchmarked 1P1D on 4+4 GB300 GPUs (one NVL72 rack) with NIXL. Setup:\n\n- NVFP4 with DSpark on both sides, CUDA graphs everywhere, no prefix caching;\n- `vllm bench serve` with fixed-length prompts: 320 unique single-turn chat prompts per input length,\neach exactly 8,192 / 32,768 / 60,000 tokens. Each prompt is a natural-text passage (WikiText-2 and\nGSM8K problems) followed by a question, with no shared prefix between prompts, sent as token IDs;\n- 1,024 output tokens per request (greedy, EOS ignored);\n- a concurrency sweep and a fixed-QPS sweep.\n\nAll configurations scored 0.965–0.980 on GSM8K-200 through the proxy, with zero failed NIXL transfers and zero failed requests.\n\n*Output tok/s per GPU vs tok/s per user, 1P1D on 4+4 GB300. Solid lines are concurrency sweeps; hollow/dotted markers are fixed-QPS sweeps.*\n\nBest output tok/s per GPU (8 GPUs) at a given minimum per-user speed:\n\n| P → D | 8k, ≥40 tok/s/user | 8k, ≥100 | 32k, ≥40 | 32k, ≥100 | 60k, ≥40 / ≥100 | c1 TTFT 8k / 32k / 60k | \n|---|---|---|---|---|---|---|\n| P DP4/EP → D DP4/EP | **718.2** | 176.7 | 202.6 | 139.2 | **108.8** | 0.49 / 1.90 / 3.65 s | \n| P DP4/EP → D TP4 | 459.2 | **194.2** | **205.2** | **150.0** | 107.2 | 0.46 / 1.88 / 3.66 s | \n| P PCP4×DCP4 → D TP4 | 350.6 | 192.6 | 91.5 | 91.5 | 49.8 | **0.35 / 1.37 / 2.57 s** | \n| P TP4 → D TP4 | 259.2 | 185.3 | 62.8 | 62.8 | 32.3 | 0.52 / 2.07 / 4.02 s | \n\n## AMD\n\nMistral Large 4 runs on AMD Instinct MI350 Series GPUs from day 0 with the ROCm build of vLLM. The sparse MLA runs on AITER’s MLA decode kernels and the DSA indexer on AITER’s gfx950 indexer.\n\n**MXFP4 experts.** On ROCm the routed experts run as OCP MXFP4 on AITER’s fused MoE. There are two ways to get them:\n\n- **AMD Quark MXFP4 checkpoint** ([Mistral-Large-4-1T-A52B-Quark-MXFP4](https://huggingface.co/amd/Mistral-Large-4-1T-A52B-Quark-MXFP4) ). Quark re-quantizes only the routed\nexperts, with static weight and dynamic activation scales. Attention, the shared experts, the dense feed-forward layers, the\nrouter, the output head, the vision encoder and the DSpark drafter keep their original precision. The checkpoint shrinks\nfrom about 1,061 GB to 578 GB, so it serves on four MI350 Series GPUs (288 GB each). The serving commands, evaluation results\nand the reproduction recipe are on the model card.\n- **Re-quantize the FP8 checkpoint at load** (`VLLM_FP8_MOE_REQUANT=mxfp4` ). The experts are converted layer by layer while\nloading, so peak memory stays low enough for TP4 as well. Expert weights drop from about 125 to 69 GiB per GPU at TP8, and\noutput throughput rises 22–71% over serving the FP8 experts at 1–64 concurrent requests.`VLLM_FP8_MOE_REQUANT_ACT` sets\nthe activation format the MXFP4 experts compute with:`a8w4` (FP8 activations, the default) or`a4w4` (MXFP4 activations).\n\nEither way, one 8-GPU node serves two TP4 replicas, each with about 6× the HBM left for KV cache and activations that the FP8 experts would leave on the same four GPUs.\n\n**Features.** TP8 and TP4, expert parallelism, image inputs through the Pixtral encoder, and DSpark speculative decoding with\nthe checkpoint’s built-in drafter are supported on ROCm. On MI350X at TP4 with the FP8 KV cache, DSpark accepts 3.1 tokens\nper step on real prompts and lowers per-token decode latency 2.7× at one request, 1.9× at 8 and 1.8× at 32 (output\nthroughput 2.1×, 1.6× and 1.4×). On ROCm the latent KV cache is FP8 (E4M3) by default, which gives\n1.86× the KV capacity of bf16; `--kv-cache-dtype bfloat16` opts out.\n\n**AITER.** `VLLM_ROCM_USE_AITER=1` runs vLLM’s hot paths on AITER, AMD’s kernel library: the fused MoE (the MXFP4 experts\non two-stage FlyDSL kernels), the block-FP8 and bf16 GEMMs, the MLA decode kernels, RMSNorm and the tensor-parallel custom\nall-reduce. The ROCm image ships AITER tuning tables for this model’s MoE and GEMM shapes, which AITER merges with its own at\nstartup; at TP4 they raise output throughput 9–16% at 1–8 concurrent requests. Two more settings help on top:\n`VLLM_ROCM_USE_AITER_TRITON_SPARSE_MLA=1` runs the sparse MLA on AITER’s Triton kernel, and\n`AITER_CUSTOM_AR_MAX_SIZE=134217728` keeps the all-reduces of large prefill steps (up to 128 MiB) on AITER’s custom\nall-reduce. Together they raise output throughput a further 2–5% at TP4 and lower time to first token 5–7%. Both commands\nbelow set all three.\n\n**Specialized decode.** For latency-bound serving, `--additional-config '{\"decode_backend\": \"latency_specialized\"}'` runs\nevery MoE layer of a 1-, 2- or 4-request decode step as one persistent FlyDSL kernel launch. The launch fuses attention, the\nindexer, the router, the MXFP4 experts and both tensor-parallel all-reduces. On MI350X at TP4 it lowers per-token decode\nlatency 1.5× at one request and 1.1× at four compared with vLLM’s standard decode on the same GPUs. It composes with\nDSpark: a single request’s verify block runs on the fused kernels, for 2.9× lower per-token latency than the standard decode\nwithout speculation. Larger batches and prefill run the standard path. It runs on the default FP8 KV cache, requires TP4 and\n`VLLM_FP8_MOE_REQUANT_ACT=a4w4`, and refuses to start otherwise, giving the reason. The kernel is adapted from ROCm\nATOM’s GLM-5.2 fused decode.\n\n**Serving on MI350 Series.** Both commands enable AITER and re-quantize the FP8 experts to MXFP4 at load:\n\n```\n# TP8, one replica per node\nVLLM_ROCM_USE_AITER=1 VLLM_FP8_MOE_REQUANT=mxfp4 VLLM_FP8_MOE_REQUANT_ACT=a8w4 \\\nVLLM_ROCM_USE_AITER_TRITON_SPARSE_MLA=1 AITER_CUSTOM_AR_MAX_SIZE=134217728 \\\nvllm serve mistralai/Mistral-Large-4-1T-A52B --tensor-parallel-size 8 \\\n  --speculative-config '{\"method\": \"dspark\", \"num_speculative_tokens\": 3}' \\\n  --tool-call-parser mistral --enable-auto-tool-choice --reasoning-parser mistral\n\n# TP4 with the specialized decode, for few concurrent users; two replicas per node (second on GPUs 4-7)\nVLLM_ROCM_USE_AITER=1 VLLM_FP8_MOE_REQUANT=mxfp4 VLLM_FP8_MOE_REQUANT_ACT=a4w4 \\\nVLLM_ROCM_USE_AITER_TRITON_SPARSE_MLA=1 AITER_CUSTOM_AR_MAX_SIZE=134217728 \\\nvllm serve mistralai/Mistral-Large-4-1T-A52B --tensor-parallel-size 4 \\\n  --additional-config '{\"decode_backend\": \"latency_specialized\"}' --max-num-seqs 64 \\\n  --speculative-config '{\"method\": \"dspark\", \"num_speculative_tokens\": 3}' \\\n  --tool-call-parser mistral --enable-auto-tool-choice --reasoning-parser mistral\n```\n\n### ROCm implementation details\n\n**Sparse MLA on AITER.** On ROCm, ML4’s sparse MLA runs on the `ROCM_AITER_MLA_SPARSE` backend. The indexer’s top-2048\npositions are converted to global cache slots, and each query token becomes its own entry in the slot stream. AITER’s\nMLA decode kernel then reads that stream in persistent mode, with work-split metadata built ahead of time to balance the\nload across CUs. AITER’s MLA kernels take a multiple of 16 heads: TP4 gives exactly 16 of ML4’s 64 heads per GPU, while\nTP8’s 8 are padded to 16. `VLLM_ROCM_USE_AITER_TRITON_SPARSE_MLA=1` switches to AITER’s Triton sparse-MLA kernel instead,\nwhich needs no head padding and no separate FP8 copy of the query. NoPE layers on the bf16 cache have no RoPE part for\nAITER’s kernel to read, so they run on vLLM’s ragged Triton sparse-attention kernels.\n\n**DSA indexer on AITER.** A fused Triton kernel quantizes each indexer key to FP8 with one scale and writes it into the\nindexer cache, 132 bytes per token per indexer layer. Within a cache block, keys are stored in 16×16 tiles, the layout\nAITER’s paged reader expects. A single layout rule is shared by the writer and every reader (insert, prefill gather and\npaged logits), so they cannot disagree. Prefill scores run on AITER’s `fp8_mqa_logits` and decode scores on AITER’s paged\nMQA-logits kernel, both reading FP8 keys. The top-2048 comes from AITER’s indexer top-k where it supports the shape, and\nfrom vLLM’s `top_k_per_row` otherwise. Shared-indexer layers reuse the top-k of the nearest earlier indexer layer, as on\nother platforms.\n\n**FP8 latent cache.** ROCm has no `fp8_ds_mla` layout, so when the checkpoint asks for an FP8 KV cache, ROCm uses a plain\nE4M3 cache instead. All 576 latent dimensions (512 NoPE and 64 RoPE) are stored as FP8 with one per-tensor scale, 576 bytes\nper token per layer, against 656 for `fp8_ds_mla` and 1,152 for bf16. The query is quantized to FP8 too, so AITER’s kernel\nruns FP8 × FP8. With 63 cache-owning layers plus the indexer, a token takes 42,096 bytes, against 78,384 in bf16: the 1.86×\nKV capacity in the AMD section.\n\n**MXFP4 experts at load, one layer at a time.** With `VLLM_FP8_MOE_REQUANT=mxfp4`, the checkpoint loader is unchanged: it\nstill loads block-FP8 expert weights with their block scales. Their parameters start on the meta device, and the layerwise\nloader materializes one layer at a time. As soon as a layer’s experts are loaded, each expert is dequantized to bf16 and\nre-quantized to MXFP4 (E2M1 values, one E8M0 scale per 32 values), and that layer’s FP8 copy is freed. Peak memory is the\nMXFP4 model plus one layer of FP8 experts, which is what lets the 1T model load on four GPUs. The weight error is about\nthe same as quantizing the bf16 weights to MXFP4 directly. The experts then run on AITER’s fused MoE: two-stage FlyDSL\nkernels with FP8 activations for `a8w4`, or MXFP4 × MXFP4 for `a4w4`.\n\n**Specialized decode: one persistent kernel per layer.** With `decode_backend=latency_specialized`, a pure decode step of\n1, 2 or 4 requests (or a single request’s DSpark verify block) runs each of the 71 MoE layers as one FlyDSL kernel launch.\nThe launch uses 256 CTAs, all resident at once on the GPU’s 256 CUs. Inside it run:\n\n- the input norm, the query and latent-KV projections with SSMax and LeRoPE;\n- the whole DSA indexer: projections, key-cache write, scoring and an exact top-2048;\n- sparse MLA over the latent cache, read in place in bf16 or FP8;\n- the output projection, the softmax top-4 router, and the shared and routed MXFP4 experts.\n\nThe embedding, the dense layer 0 and the final norm stay on vLLM’s modules.\n\nBoth tensor-parallel all-reduces run inside the kernel, over peer memory between the four GPUs, so a layer is one launch with no host round-trip. The kernels are built for 1, 2 and 4 rows, which are vLLM’s full CUDA-graph capture sizes, and are replayed inside those graphs. Any other step runs vLLM’s layers: prefill, mixed batches, wider batches. The kernel is adapted from ROCm ATOM’s GLM-5.2 fused decode, extended for ML4’s layer types:\n\n- no query LoRA;\n- SSMax on the query and the index query;\n- softmax routing;\n- NoPE layers;\n- KV-sharing layers that read layer 35’s latent cache;\n- index-sharing layers.\n\n**Fail-stop.** A persistent kernel that waits on peer GPUs must never hang the server. Every in-kernel wait gives up after\n10 seconds, and giving up sets a sticky fault word on every rank, which ends the other ranks’ waits at once. After every\nforward, a device-side assert checks the word before logits are computed. If the word is set, the step’s remaining work\nnever runs, the worker aborts and the engine stops, so no token from a failed step reaches a client. The check costs\nabout 1 µs per launch.\n\nThe path is also default-deny. If a deployment uses anything the kernels don’t serve, startup fails with one error listing\nevery reason, and all TP ranks agree on each check before anything collective runs. Unsupported features include other\nTP sizes, expert or context parallelism, LoRA, KV connectors, other speculative methods, and experts not stored as `a4w4`.\n\n## TPU\n\nMistral Large 4 runs on **TPU v7x from day 0** with `vllm-torchtpu`, including support for the\nPixtral vision encoder. Our validated configuration combines intra-host tensor parallelism\nwith cross-host data parallelism with expert parallelism across 16 TPU v7x chips.\n\n**Serving on TPU v7x:** The following command starts a single data parallel worker on one host:\n\n```\nvllm serve mistralai/Mistral-Large-4-1T-A52B \\\n  --tensor-parallel-size 8 --data-parallel-size 4 --data-parallel-size-local 1 \\\n  --data-parallel-start-rank \"$TPU_WORKER_ID\" \\\n  --data-parallel-address \"$HEAD_IP\" --data-parallel-rpc-port 29696 \\\n  --enable-expert-parallel --enable-ep-weight-filter \\\n  --dtype bfloat16 --kv-cache-dtype fp8_e4m3\n```\n\nThe TPU implementation relies on `vllm-torchtpu`’s sparse MLA and DSA indexer Pallas kernels,\nextending MLA with cross-layer KV cache sharing. We store the MLA latent KV cache in FP8 and\nsupport FP8 weight and activation quantization for dense linear projections. Routed experts use\nweight-only FP4 quantization (W4A16).\n\n## HiSparse: host-memory sparse KV offload\n\nSparse attention reads only 2048 keys per query, so most of a long request’s KV doesn’t need\nto live in GPU memory. HiSparse keeps the full latent KV in pinned host memory. It keeps a\nresident GPU working set plus a per-request hot buffer, and copies the selected rows from host on\na miss. Enable it with the `HiSparseConnector`:\n\n```\n--kv-transfer-config '{\"kv_connector\": \"HiSparseConnector\", \"kv_role\": \"kv_both\",\n                       \"kv_connector_extra_config\": {\"host_pool_gib\": 440}}'\n```\n\nFor ML4 we made HiSparse work with **cross-layer KV sharing**. The nine consumer layers bind to\nlayer 35’s resident view and host copy and keep only their own hot buffer and LRU (each has its\nown indexer). Nothing is duplicated, and no environment variable is needed. Compared with\nmaterializing the shared layers, this holds 14.3% more tokens for the same host pool. We also fixed:\n\n- reading `index_topk` from the text config of multimodal models;\n- a double host mirror during CUDA graph capture.\n\n**Capacity.** On a single GB300 node (TP4, NVFP4), the four TP ranks share one deduplicated host\npool, so `host_pool_gib` is the total. With a 440 GiB pool, HiSparse admits 8.77M KV tokens at\nstartup and holds up to 11.25M in steady state, vs 1.97M on the GPU alone (4.5x to 5.7x). GSM8K-200\nis unchanged (0.975 both), and needle-in-a-haystack passes at ~98k (10/10) and ~248k (5/5) tokens.\n\n*KV token capacity of one TP4 server on a single GB300 node (NVFP4, max_model_len 262k, prefix caching off): 1.97M tokens on the GPU alone; with HiSparse and a 440 GiB host pool shared by the four TP ranks, 8.77M admitted at startup (4.5x) and up to 11.25M in steady state (5.7x). Layers that reuse layer 35’s KV are stored on the host only once.*\n\n## What’s next\n\n- Upstreaming: MLA cross-layer KV sharing, DSpark sliding-window MLA drafts and DSpark under DCP and HiSparse KV-sharing support.\n- A DCP decode node with DSpark in P/D, and a separate draft block pool so DSpark under DCP keeps full KV capacity.\n- HiSparse with GPU landing for NIXL imports and node-shared host pools for multi-node TP.\n- More decode fusions for low-concurrency latency.\n\n## Acknowledgement\n\nMistral Large 4 support in vLLM was a collaboration led by Nicolò Lucchesi and the Mistral Inference team. We would like to extend our gratitude to the vLLM maintainers who reviewed, designed, and landed much of the underlying work: Lucas Wilkinson, Flora Feng, Matt Bonanni, and the rest of the Red Hat team; the Inferact team as a whole; Cyrus Leung as well as the entire vLLM community.\n\nOn AMD side, we would also like to thank Hongxia Yang, Felix Marty, Andy Luo, Indira Vats for Day 0 enablement and optimization on AMD Instinct GPUs, and Bowen Bao, Chuan Li, Peng Sun for general support.", "url": "https://wpnews.pro/news/day-0-support-for-mistral-large-4-on-vllm", "canonical_source": "https://vllm-project.github.io/2026/10/29/mistral-large-4.html", "published_at": "2026-10-29 00:00:00+00:00", "updated_at": "2026-10-29 14:20:59.813446+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "machine-learning"], "entities": ["vLLM", "Mistral", "Mistral Large 4", "DSpark", "Pixtral", "AMD Instinct MI350", "ROCm", "NIXL"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/day-0-support-for-mistral-large-4-on-vllm", "markdown": "https://wpnews.pro/news/day-0-support-for-mistral-large-4-on-vllm.md", "text": "https://wpnews.pro/news/day-0-support-for-mistral-large-4-on-vllm.txt", "jsonld": "https://wpnews.pro/news/day-0-support-for-mistral-large-4-on-vllm.jsonld"}}