{"slug": "deploying-qwen3-8-2-4t-a95b-with-vllm-verified-gpu-pods-quants-and-serving", "title": "Deploying Qwen3.8-2.4T-A95B with vLLM: Verified GPU Pods, Quants, and Serving Recipes", "summary": "A developer detailed deployment recipes for Qwen3.8-2.4T-A95B, a 2.4-trillion-parameter Mixture-of-Experts model with 95B active parameters, using vLLM. The guide covers verified GPU configurations, including NVFP4 and FP8 checkpoints, and highlights that enabling Multi-Token Prediction with three speculative tokens can boost output throughput from around 130 to over 300 tokens per second per user in vLLM's tests.", "body_md": "Qwen3.8-2.4T-A95B is a 2.4-trillion-parameter Mixture-of-Experts model with roughly 95B parameters active for each token. If you're planning to self-host it, the first thing to know is that this is a genuinely large distributed model: even the low-precision checkpoints are measured in terabytes.\n\nThe official open checkpoint is:\n\n```\nQwen/Qwen3.8-2.4T-A95B\n```\n\nThe model has 512 routed experts and selects 10 of them per token alongside one shared expert. Its 92-layer backbone mixes 69 Gated DeltaNet linear-attention layers with 23 full-attention layers, with full attention appearing every fourth layer.\n\nNative context is **262,144 tokens**, with an extended configuration available up to roughly **1.01 million tokens**.\n\nThe open checkpoint is text-only and always uses reasoning. This is different from Qwen's hosted `Qwen3.8-Max`\n\nservice, which adds features such as vision input and non-thinking mode.\n\nFor GPU deployment, the main decision is not whether 2.4T parameters will somehow fit. It is **which precision format gives you a documented configuration on the hardware you actually have**.\n\nThe practical options today are:\n\n| Your GPUs | Checkpoint | Documented setup |\n|---|---|---|\n8× B300 |\n`Inferact/Qwen3.8-2.4T-A95B-NVFP4` |\nTP8 |\n8× GB300 |\n`Inferact/Qwen3.8-2.4T-A95B-NVFP4` |\nTP8 across two NVL4 trays |\n16× B300 |\n`Qwen/Qwen3.8-2.4T-A95B-FP8` |\nTP16 |\n16× GB300 |\n`Qwen/Qwen3.8-2.4T-A95B-FP8` |\nTP16 |\n12× GB300 |\n`Qwen/Qwen3.8-2.4T-A95B-FP8` |\nTP4 × PP3 |\n8× MI355X |\n`Inferact/Qwen3.8-2.4T-A95B-MXFP4` |\nTP8 |\n\nThe full BF16 checkpoint is roughly **4.45 TiB**. The official FP8 version is around **2.27 TiB**, while the NVFP4 checkpoint used in the NVIDIA eight-GPU recipe is around **1.32 TiB**.\n\nThat is why NVFP4 is the most approachable NVIDIA deployment if your goal is simply to get Qwen3.8 running without moving immediately to a 16-GPU cluster.\n\nH100, H200, A100, B200 and smaller GPU configurations are not included here. Current vLLM material contains sizing information for some of those GPUs, but not equivalent end-to-end serving recipes. This guide does not turn memory estimates into deployment claims.\n\nThe current vLLM recipe recommends a recent nightly build rather than an older stable release.\n\nCreate an environment with:\n\n```\nuv venv\nsource .venv/bin/activate\n\nuv pip install -U vllm \\\n  --extra-index-url https://wheels.vllm.ai/nightly\n\nuv pip install -U \"transformers>=5.4.0\"\n```\n\nOnce you have a working build, pin it. Qwen3.8 relies on recently added model and kernel support, so continuously upgrading a production server to whatever nightly happens to be current is unnecessary risk.\n\nFor NVIDIA, the smallest documented configuration uses:\n\n```\nInferact/Qwen3.8-2.4T-A95B-NVFP4\n```\n\nwith eight GPUs.\n\nStart with:\n\n```\nvllm serve Inferact/Qwen3.8-2.4T-A95B-NVFP4 \\\n  --tensor-parallel-size 8 \\\n  --max-model-len 262144 \\\n  --kv-cache-dtype fp8 \\\n  --reasoning-parser qwen3 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser qwen3_coder \\\n  --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}'\n```\n\nThis configuration applies to an eight-GPU B300 system and is also documented across eight GB300 GPUs spanning two NVL4 trays.\n\nvLLM's optimized NVFP4 path also uses:\n\n```\n--linear-backend flashinfer_cutedsl\n```\n\nbut that flag depends on having the matching FlashInfer environment. Add it when reproducing the corresponding vLLM container/software stack rather than assuming every vLLM installation has the required backend.\n\nQwen3.8 contains a built-in Multi-Token Prediction head. vLLM's published results show that using three speculative tokens can make a substantial difference.\n\nOn its low-latency tests:\n\n```\nFP8 TP16\nWithout MTP: 130 output tok/s/user\nMTP-3:       307 output tok/s/user\n\nNVFP4 TP8\nWithout MTP: 133 output tok/s/user\nMTP-3:       304 output tok/s/user\n```\n\nThose are measurements from vLLM's hardware and workload, not promised performance for every server. They do make MTP-3 worth testing from the beginning:\n\n```\n--speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}'\n```\n\nUsing only one speculative token performed much less convincingly in the same testing. vLLM measured 64.8% acceptance for MTP-1, and at higher concurrency the extra speculative work could actually hurt throughput.\n\nIf you want Qwen's official FP8 checkpoint:\n\n```\nQwen/Qwen3.8-2.4T-A95B-FP8\n```\n\nthe documented low-latency configuration moves to **16 GPUs**.\n\nOn B300, that normally means two eight-GPU nodes using TP16.\n\nThe head node runs:\n\n```\nvllm serve Qwen/Qwen3.8-2.4T-A95B-FP8 \\\n  --tensor-parallel-size 16 \\\n  --nnodes 2 \\\n  --node-rank 0 \\\n  --master-addr $HEAD_ADDR \\\n  --max-model-len 262144 \\\n  --kv-cache-dtype fp8 \\\n  --reasoning-parser qwen3 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser qwen3_coder \\\n  --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}'\n```\n\nThe second node uses the same distributed configuration with:\n\n```\n--node-rank 1 \\\n--headless\n```\n\nOnly rank 0 should expose the API server.\n\nThere is also a verified **12× GB300** FP8 configuration using:\n\n```\nTP4 × PP3\n```\n\nWhy not TP12? Qwen3.8 has 64 full-attention heads, so its tensor-parallel size needs to divide 64. TP12 does not.\n\nvLLM verified the TP4 × PP3 layout on 12 GB300 GPUs, including model loading, CUDA graph capture and generation. It is a useful option when you have three four-GPU GB300 trays, although the ordinary TP8 and TP16 setups remain simpler.\n\nThe documented AMD route uses:\n\n```\nInferact/Qwen3.8-2.4T-A95B-MXFP4\n```\n\non **8× MI355X**.\n\n```\nvllm serve Inferact/Qwen3.8-2.4T-A95B-MXFP4 \\\n  --tensor-parallel-size 8 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser qwen3_coder \\\n  --reasoning-parser qwen3 \\\n  --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}'\n```\n\nOne difference from the NVIDIA command is worth preserving: the current vLLM recipe does not recommend blindly adding FP8 KV cache to this ROCm setup.\n\nDo not assume:\n\n```\n--kv-cache-dtype fp8\n```\n\nworks with every ROCm/vLLM combination. Add it only after confirming it on the exact software build running on the node.\n\nAMD also publishes:\n\n```\namd/Qwen3.8-2.4T-A95B-Quark-MXFP4\n```\n\nfor MI350 and MI355 hardware.\n\nIn that conversion, routed experts use OCP MXFP4 while components such as attention, routers, the shared expert, LM head and MTP layer remain at higher precision.\n\nAMD's published GSM8K reproduction reported:\n\n```\nFP8 baseline: 97.49\nMXFP4:        97.49\n```\n\nThat result was reproduced with SGLang at TP8 on MI35x hardware. It is evidence for the quality of AMD's quantization, not a vLLM throughput benchmark.\n\nThe open checkpoint natively supports:\n\n```\n262,144 tokens\n```\n\nand can be extended to about:\n\n```\n1,010,000 tokens\n```\n\nwith vLLM:\n\n```\nexport VLLM_ALLOW_LONG_MAX_MODEL_LEN=1\n```\n\nand:\n\n```\n--max-model-len 1010000 \\\n--hf-overrides '{\"max_position_embeddings\":1010000}'\n```\n\nThere is little reason to enable 1M context automatically.\n\nLong context consumes memory that could otherwise be used for concurrent requests. vLLM's own NVFP4 testing illustrates the tradeoff: a configuration sized around the 262K window could hold roughly 25 concurrent requests in the available KV-cache budget, while a much shorter workload around 8K input plus 1K output allowed hundreds.\n\nIf you're serving repository-scale coding agents, very long context may be worth the cost. If most requests are 10K or 20K tokens, reserving for one million tokens mostly reduces how many users the GPUs can serve at once.\n\nSet:\n\n```\n--max-model-len\n```\n\nfor the workload you actually expect.\n\nQwen3.8 is a reasoning model. The open checkpoint does not expose a normal non-thinking mode.\n\nIt supports three reasoning-effort settings:\n\n```\nxhigh\nmedium\nlow\n```\n\n`xhigh`\n\nis the default.\n\nQwen also preserves thinking across turns by default, which matters for agentic sessions where earlier reasoning and tool interactions are part of the ongoing context.\n\nStart vLLM with:\n\n```\n--reasoning-parser qwen3 \\\n--enable-auto-tool-choice \\\n--tool-call-parser qwen3_coder\n```\n\nQwen's recommended generation settings include:\n\n```\ntemperature = 1.0\ntop_p = 0.95\ntop_k = 20\nmin_p = 0.0\npresence_penalty = 0.0\nrepetition_penalty = 1.0\n```\n\nA normal OpenAI-compatible client looks like:\n\n``` python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"http://localhost:8000/v1\",\n    api_key=\"EMPTY\",\n    timeout=3600,\n)\n\nresponse = client.chat.completions.create(\n    model=\"Inferact/Qwen3.8-2.4T-A95B-NVFP4\",\n    messages=[\n        {\n            \"role\": \"user\",\n            \"content\": \"Review this distributed queue design and identify its failure modes.\"\n        }\n    ],\n    reasoning_effort=\"medium\",\n    temperature=1.0,\n    top_p=0.95,\n    max_tokens=16384,\n)\n\nprint(response.choices[0].message)\n```\n\nFor hard coding and agent tasks, leave enough output budget for the reasoning trace. An aggressive `max_tokens`\n\nlimit can terminate the generation before the model reaches its final answer.\n\nAt Qwen3.8's scale, startup time becomes part of operating the server.\n\nvLLM's NVFP4 testing found that:\n\n```\n--load-format fastsafetensors \\\n--safetensors-load-strategy lazy\n```\n\nreduced model load time from **545 seconds to 306 seconds** on the shared storage used in that test.\n\nThat result will vary with storage performance, but persistent model storage is clearly preferable to downloading or repeatedly copying a terabyte-scale checkpoint whenever a machine restarts.\n\nvLLM also uses a larger engine startup timeout:\n\n```\nexport VLLM_ENGINE_READY_TIMEOUT_S=3600\n```\n\nAnd don't make your own readiness decision solely from whether the process is alive. Send a small request to:\n\n```\n/v1/chat/completions\n```\n\nand verify that the model can actually generate.\n\nIf you're renting GPUs specifically for Qwen3.8-2.4T-A95B, the current documented choices are fairly straightforward.\n\n**8× B300 + NVFP4** is the simplest NVIDIA configuration in the current vLLM recipe.\n\n**8× GB300 + NVFP4** gives you the equivalent Grace Blackwell path across two NVL4 trays.\n\nUse **16× B300 or GB300 + FP8** when you specifically want the official FP8 checkpoint and have enough hardware for TP16.\n\nIf you have three GB300 trays, **12× GB300 FP8 with TP4 × PP3** is an upstream-verified alternative.\n\nOn AMD, use **8× MI355X + MXFP4**.\n\nH100, H200, A100, B200 and smaller GPU counts are intentionally not recommended here. There may be enough aggregate memory in some of those configurations, but the current sources do not provide the same reproducible end-to-end deployment evidence.\n\nFor Qwen3.8, choosing the GPUs is only part of the deployment. The exact quant, tensor-parallel layout, KV-cache budget, vLLM build, network topology and MTP configuration all affect whether the resulting server is useful once real traffic starts hitting it.\n\n**Qwen3.8-2.4T-A95B official model card**\n\n[https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B)\n\n**Official FP8 checkpoint**\n\n[https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8)\n\n**Official Qwen3.8-2.4T-A95B deployment recipe**\n\n[https://recipes.vllm.ai/Qwen/Qwen3.8-2.4T-A95B](https://recipes.vllm.ai/Qwen/Qwen3.8-2.4T-A95B)\n\n**vLLM Qwen3.8 launch notes and benchmarks**\n\n[https://vllm.ai/blog/2026-08-12-qwen3.8](https://vllm.ai/blog/2026-08-12-qwen3.8)\n\n**Qwen3.8 NVFP4 checkpoint**\n\n[https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-NVFP4](https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-NVFP4)\n\n**Qwen3.8 MXFP4 checkpoint**\n\n[https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-MXFP4](https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-MXFP4)", "url": "https://wpnews.pro/news/deploying-qwen3-8-2-4t-a95b-with-vllm-verified-gpu-pods-quants-and-serving", "canonical_source": "https://dev.to/nick_k_gpus_market/deploying-qwen38-24t-a95b-with-vllm-verified-gpu-pods-quants-and-serving-recipes-g8a", "published_at": "2026-08-15 03:19:40+00:00", "updated_at": "2026-08-15 03:41:26.981305+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "mlops", "developer-tools"], "entities": ["Qwen", "vLLM", "NVIDIA", "B300", "GB300", "MI355X", "Inferact", "FlashInfer"], "alternates": {"html": "https://wpnews.pro/news/deploying-qwen3-8-2-4t-a95b-with-vllm-verified-gpu-pods-quants-and-serving", "markdown": "https://wpnews.pro/news/deploying-qwen3-8-2-4t-a95b-with-vllm-verified-gpu-pods-quants-and-serving.md", "text": "https://wpnews.pro/news/deploying-qwen3-8-2-4t-a95b-with-vllm-verified-gpu-pods-quants-and-serving.txt", "jsonld": "https://wpnews.pro/news/deploying-qwen3-8-2-4t-a95b-with-vllm-verified-gpu-pods-quants-and-serving.jsonld"}}