How to Run Qwen3.8-Flash-Next FP4 Locally (It Needs Blackwell and 55 Env Vars)

NVIDIA released an FP4 build of Alibaba's Qwen3.8-Flash-Next MoE model that shrinks the checkpoint from 355 GB in BF16 to roughly 96 GiB, using quantization-aware distillation with Model Optimizer v0.47.0 rather than plain post-training quantization. The build requires Blackwell hardware and a launch script exporting 55 environment variables, and NVIDIA reports a 0.7-point drop on GPQA Diamond (92.2% to 91.5%) and ToolEvalBench (89.4% to 88.7%). Because 96 GiB of weights leaves no KV cache room on a 96 GB RTX PRO 6000, NVIDIA's validated configuration offloads the n-gram embedding table to CPU to reach 850K tokens of FP8 KV cache.

NVIDIA's FP4 build of Qwen3.8-Flash-Next is 96 GiB on disk. The BF16 original is about 355 GB. NVIDIA says the accuracy cost of that shrink is 0.7 points on GPQA Diamond. If you want to run Qwen3.8 Flash Next locally, that sounds like the obvious download. It isn't, at least not yet. The first line of the model card tells you upstream vLLM can't load it. Blackwell is the only supported architecture, and the RTX PRO 6000 launch script exports 55 environment variables before it gets to vllm serve . This post covers what NVIDIA changed, how the size stacks up against GGUF, which benchmark numbers hold up, and the run commands for RTX PRO 6000 and DGX Spark. My sources are the NVIDIA model card https://huggingface.co/nvidia/Qwen3.8-Flash-Next-FP4-Local , the Qwen card https://huggingface.co/Qwen/Qwen3.8-Flash-Next , Unsloth's and bartowski's GGUF pages, and the community speed thread. I haven't run this checkpoint myself, so every number here comes from those pages, linked inline. The base model is Alibaba's Qwen3.8-Flash-Next, released on August 26, 2026 Qwen blog https://qwen.ai/blog?id=qwen3.8-flash-next . It's a MoE with 125B total parameters and 6B active, plus a 51B n-gram embedding table the card calls it PLE and a 4B MTP head. It has a vision encoder and hybrid attention Gated DeltaNet plus Qwen Sparse Attention . Native context is 262K tokens, extendable to 1M. NVIDIA published nvidia/Qwen3.8-Flash-Next-FP4-Local on 10/05/2026. The most interesting part of the card is how it was made. Instead of a plain post-training quant, NVIDIA used quantization-aware distillation QAD with Model Optimizer v0.47.0. The BF16 model acts as teacher, and the low-precision weights are trained to match its token probabilities and hidden states. There are even separate losses for end-of-thinking and end-of-response behavior, which the card says target the teacher's decisions to continue, finish reasoning and stop responding. To me that reads like a direct fix for the quantized-model habit of thinking forever or never stopping. The training budget was 500M tokens over 3 epochs: 2,500 trunk updates, then 1,500 joint-refinement updates that also train the n-gram tables. Not everything ends up as FP4: | Component | Precision | |---|---| | Decoder routed experts | NVFP4 | | Decoder attention | MXFP8 | | MTP routed experts | NVFP4 | | Vision attention linears | MXFP8 | | Vision MLP and merger | NVFP4 | | Other vision tensors, norms | BF16 / FP32 | So attention stays at 8-bit and the experts carry the FP4 savings. That's a sensible split, because routed experts are where the bytes are. Sizes as published. These aren't measurements of mine. | Build | Size | Source | |---|---|---| | BF16 original | 355 GB | Unsloth docs https://unsloth.ai/docs/models/qwen3.8-next | | GGUF Q8 0 | 188.26 GB | bartowski https://huggingface.co/bartowski/Qwen3.8-Flash-Next-GGUF | | GGUF Q4 K M | 119.60 GB | bartowski | | GGUF UD-Q4 K XL | 111.3 GB | Unsloth | | GGUF IQ4 XS | 97.68 GB | bartowski | | NVIDIA FP4-Local | ~96 GiB | NVIDIA card | | GGUF UD-IQ1 S | 72.5 GB | Unsloth | One unit catch: 96 GiB is about 103 GB. That puts the NVIDIA build between IQ4 XS and UD-Q4 K XL, so it isn't smaller than every 4-bit GGUF, as the "96" suggests. It's even bigger than bartowski's IQ4 NL 100.29 GB . If you're picking it, pick it for the training method, because the file size alone doesn't win. Do the arithmetic: 96 GiB of weights on a 96 GB RTX PRO 6000 leaves no room for KV cache. NVIDIA's validated config offloads the n-gram embedding table to CPU VLLM PLE CPU OFFLOAD=1 and claims 850K tokens of FP8 KV cache that way. On DGX Spark the card says it runs with or without PLE offload, with 1M tokens of FP8 KV cache when the table goes to disk. | Precision | GPQA Diamond | ToolEvalBench | |---|---|---| | BF16 | 92.2% | 89.4% | | FP4-Local | 91.5% | 88.7% | Both drop by 0.7 points. I wouldn't call that lossless yet. GPQA Diamond has 198 questions, so 0.7 points is roughly 1.4 questions. That's within the noise of a single sampled run, and it cuts both ways: the loss is small, and the benchmark can't prove it's small. NVIDIA's BF16 baseline is 92.2, but Qwen's own card https://huggingface.co/Qwen/Qwen3.8-Flash-Next reports 91.7 for the same model, presumably from a different harness or settings. Compare FP4 against NVIDIA's BF16 number, not Qwen's, or you'll conclude the quant only lost 0.2 points. Then there's vision. The card lists image and video input, quantizes the vision MLP to NVFP4, and then publishes zero vision benchmarks. The distillation data is text only. If you picked this build for image input, you're the eval. For comparison, Unsloth publishes KL divergence for its quants: UD-Q4 K XL has a mean KLD of 0.046893 and 92.255% top-1 agreement with BF16. That's a different metric from NVIDIA's, so the published numbers don't let you call a winner either way. Blackwell plus vLLM, and only in community containers. The original Qwen weights, by contrast, run on vLLM, SGLang, Transformers and TokenSpeed https://huggingface.co/Qwen/Qwen3.8-Flash-Next , and SGLang had day-0 support https://www.lmsys.org/blog/2026-08-26-qwen-flash-next/ . NVIDIA's build also isn't the only NVFP4 checkpoint. The LMSYS post points to RadixArk/Qwen3.8-Flash-Next-NVFP4 https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4 , an SGLang checkpoint that LMSYS says was available the same day as the model. It's a plain post-training quant: routed experts only, calibrated on 128 CNN/DailyMail articles truncated to 512 tokens, validated on GB300 and B300. LMSYS reports 540 tok/s at batch size 1 with MTP at TP4 on B200. That's datacenter hardware, but it shows what NVIDIA's QAD run is competing with: 128 short calibration samples on one side, 500M tokens of distillation on the other. This is the simplified one-liner from the card, copied as published: docker run -d --name qwen38-tp1 --init --gpus '"device=0"' \ --restart on-failure --network host --ipc host --shm-size 32g --stop-timeout 30 \ --ulimit memlock=-1 --ulimit stack=67108864:67108864 \ -v model-cache:/root/.cache/huggingface -v qwen38-runtime:/cache \ -e PROFILE=qwen38-flash-next -e PORT=8000 \ ghcr.io/local-inference-lab/vllm:release-10152026 The verbose path on the card is the one to read even if you use the one-liner, because it shows what the profile turns on: --quantization modelopt mixed , --kv-cache-dtype fp8 , --max-model-len 262144 , --max-num-seqs 16 , --gpu-memory-utilization 0.975 , and MTP speculative decoding with "num speculative tokens":3 . A memory utilization of 0.975 leaves almost nothing for your desktop, so plan on a headless box or a second GPU for display. Start the container set HF TOKEN on the host first : mkdir -p $HOME/.cache/huggingface $HOME/.cache/vllm $HOME/.cache/flashinfer $HOME/.triton $HOME/.cache/b12x docker run --rm -it \ --gpus all \ --ipc host \ --network host \ --shm-size=16g \ --ulimit memlock=-1 \ --ulimit stack=67108864 \ --security-opt seccomp=unconfined \ --security-opt apparmor=unconfined \ -e HF TOKEN=$HF TOKEN \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -v $HOME/.cache/vllm:/root/.cache/vllm \ -v $HOME/.cache/flashinfer:/root/.cache/flashinfer \ -v $HOME/.triton:/root/.triton \ -v $HOME/.cache/b12x:/root/.cache/b12x \ --entrypoint /bin/bash \ eugr/spark-vllm-b12x:latest Inside it, the card exports 14 variables including VLLM PLE TABLE MEMORY="disk" and CUTE DSL ARCH="sm 121a" and then runs vllm serve nvidia/Qwen3.8-Flash-Next-FP4-Local with --load-format b12x , --max-num-seqs 8 , --gpu-memory-utilization 0.8 and MTP at 4 speculative tokens. Copy that block from the card verbatim instead of retyping it. Both configs use --reasoning-parser qwen3 and --tool-call-parser qwen3 xml , so tool calling works through the OpenAI-compatible API on port 8000. Qwen recommends temperature=1.0, top p=0.95, top k=20 for thinking mode. GGUF with llama.cpp. Unsloth's run command: unsloth run --model unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4 K XL The people in the Unsloth speed thread https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/discussions/3 aren't on Blackwell workstations. One reports 19 t/s decode on an RTX 3090 with 128 GB DDR4 at Q4 K XL. Another gets 39.48 t/s at depth 0 on an M1 Ultra 128GB at IQ4 NL, using a llama.cpp branch with Metal optimizations. On an RTX 6000 PRO, one user reports 59.70 t/s TG at Q4 K XL, and Unsloth's docs quote 170 t/s with MTP on one RTX 6000 PRO against a 100 t/s baseline, without naming the quant. NVIDIA publishes no throughput numbers for the FP4 build at all, which is a strange omission for a release aimed at local inference. If you own an RTX PRO 6000 or a DGX Spark, try the NVIDIA build. QAD is a real method, and a 0.7-point drop is about as good as a 4-bit-class quant gets. Everyone else should stick with GGUF, which runs in mainline llama.cpp on the hardware you already have without a single exported env var. Re-check once upstream vLLM merges the scale format, because the 55-variable setup is the part I'd least want to maintain. So: do you trust a vendor FP4 quant over community GGUFs from unsloth and bartowski? Has anyone run FP4-Local and a Q4 K XL side by side on the same prompts and seen a quality difference? Post your numbers, especially on image input, since NVIDIA didn't.