How to Run Qwen3.8-Flash-Next FP4 Locally (It Needs Blackwell and 55 Env Vars)

NVIDIA's FP4 build of Qwen3.8-Flash-Next is 96 GiB on disk. The BF16 original is about 355 GB. NVIDIA says the accuracy cost of that shrink is 0.7 points on GPQA Diamond. If you want to run Qwen3.8 Flash Next locally, that sounds like the obvious download.

It isn't, at least not yet. The first line of the model card tells you upstream vLLM can't load it. Blackwell is the only supported architecture, and the RTX PRO 6000 launch script exports 55 environment variables before it gets to vllm serve.

This post covers what NVIDIA changed, how the size stacks up against GGUF, which benchmark numbers hold up, and the run commands for RTX PRO 6000 and DGX Spark. My sources are the NVIDIA model card, the Qwen card, Unsloth's and bartowski's GGUF pages, and the community speed thread. I haven't run this checkpoint myself, so every number here comes from those pages, linked inline.

The base model is Alibaba's Qwen3.8-Flash-Next, released on August 26, 2026 (Qwen blog). It's a MoE with 125B total parameters and 6B active, plus a 51B n-gram embedding table (the card calls it PLE) and a 4B MTP head. It has a vision encoder and hybrid attention (Gated DeltaNet plus Qwen Sparse Attention). Native context is 262K tokens, extendable to 1M.

NVIDIA published nvidia/Qwen3.8-Flash-Next-FP4-Local on 10/05/2026. The most interesting part of the card is how it was made. Instead of a plain post-training quant, NVIDIA used quantization-aware distillation (QAD) with Model Optimizer v0.47.0. The BF16 model acts as teacher, and the low-precision weights are trained to match its token probabilities and hidden states. There are even separate losses for end-of-thinking and end-of-response behavior, which the card says target the teacher's decisions to continue, finish reasoning and stop responding. To me that reads like a direct fix for the quantized-model habit of thinking forever or never stopping. The training budget was 500M tokens over 3 epochs: 2,500 trunk updates, then 1,500 joint-refinement updates that also train the n-gram tables.

Not everything ends up as FP4:

So attention stays at 8-bit and the experts carry the FP4 savings. That's a sensible split, because routed experts are where the bytes are.

Sizes as published. These aren't measurements of mine.

One unit catch: 96 GiB is about 103 GB. That puts the NVIDIA build between IQ4_XS and UD-Q4_K_XL, so it isn't smaller than every 4-bit GGUF, as the "96" suggests. It's even bigger than bartowski's IQ4_NL (100.29 GB). If you're picking it, pick it for the training method, because the file size alone doesn't win.

Do the arithmetic: 96 GiB of weights on a 96 GB RTX PRO 6000 leaves no room for KV cache. NVIDIA's validated config offloads the n-gram embedding table to CPU (VLLM_PLE_CPU_OFFLOAD=1) and claims 850K tokens of FP8 KV cache that way. On DGX Spark the card says it runs with or without PLE offload, with 1M tokens of FP8 KV cache when the table goes to disk.

Both drop by 0.7 points. I wouldn't call that lossless yet.

GPQA Diamond has 198 questions, so 0.7 points is roughly 1.4 questions. That's within the noise of a single sampled run, and it cuts both ways: the loss is small, and the benchmark can't prove it's small.

NVIDIA's BF16 baseline is 92.2, but Qwen's own card reports 91.7 for the same model, presumably from a different harness or settings. Compare FP4 against NVIDIA's BF16 number, not Qwen's, or you'll conclude the quant only lost 0.2 points.

Then there's vision. The card lists image and video input, quantizes the vision MLP to NVFP4, and then publishes zero vision benchmarks. The distillation data is text only. If you picked this build for image input, you're the eval.

For comparison, Unsloth publishes KL divergence for its quants: UD-Q4_K_XL has a mean KLD of 0.046893 and 92.255% top-1 agreement with BF16. That's a different metric from NVIDIA's, so the published numbers don't let you call a winner either way.

Blackwell plus vLLM, and only in community containers.

The original Qwen weights, by contrast, run on vLLM, SGLang, Transformers and TokenSpeed, and SGLang had day-0 support.

NVIDIA's build also isn't the only NVFP4 checkpoint. The LMSYS post points to RadixArk/Qwen3.8-Flash-Next-NVFP4, an SGLang checkpoint that LMSYS says was available the same day as the model. It's a plain post-training quant: routed experts only, calibrated on 128 CNN/DailyMail articles truncated to 512 tokens, validated on GB300 and B300. LMSYS reports 540 tok/s at batch size 1 with MTP at TP4 on B200. That's datacenter hardware, but it shows what NVIDIA's QAD run is competing with: 128 short calibration samples on one side, 500M tokens of distillation on the other.

This is the simplified one-liner from the card, copied as published:

  docker run -d --name qwen38-tp1 --init --gpus '"device=0"' \
    --restart on-failure --network host --ipc host --shm-size 32g --stop-timeout 30 \
    --ulimit memlock=-1 --ulimit stack=67108864:67108864 \
    -v model-cache:/root/.cache/huggingface -v qwen38-runtime:/cache \
    -e PROFILE=qwen38-flash-next -e PORT=8000 \
    ghcr.io/local-inference-lab/vllm:release-10152026

The verbose path on the card is the one to read even if you use the one-liner, because it shows what the profile turns on: --quantization modelopt_mixed, --kv-cache-dtype fp8, --max-model-len 262144, --max-num-seqs 16, --gpu-memory-utilization 0.975, and MTP speculative decoding with "num_speculative_tokens":3. A memory utilization of 0.975 leaves almost nothing for your desktop, so plan on a headless box or a second GPU for display.

Start the container (set HF_TOKEN on the host first):

mkdir -p $HOME/.cache/huggingface $HOME/.cache/vllm $HOME/.cache/flashinfer $HOME/.triton $HOME/.cache/b12x

docker run --rm -it \
  --gpus all \
  --ipc host \
  --network host \
  --shm-size=16g \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  --security-opt seccomp=unconfined \
  --security-opt apparmor=unconfined \
  -e HF_TOKEN=$HF_TOKEN \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v $HOME/.cache/vllm:/root/.cache/vllm \
  -v $HOME/.cache/flashinfer:/root/.cache/flashinfer \
  -v $HOME/.triton:/root/.triton \
  -v $HOME/.cache/b12x:/root/.cache/b12x \
  --entrypoint /bin/bash \
  eugr/spark-vllm-b12x:latest

Inside it, the card exports 14 variables (including VLLM_PLE_TABLE_MEMORY="disk" and CUTE_DSL_ARCH="sm_121a") and then runs vllm serve nvidia/Qwen3.8-Flash-Next-FP4-Local with --load-format b12x, --max-num-seqs 8, --gpu-memory-utilization 0.8 and MTP at 4 speculative tokens. Copy that block from the card verbatim instead of retyping it.

Both configs use --reasoning-parser qwen3 and --tool-call-parser qwen3_xml, so tool calling works through the OpenAI-compatible API on port 8000. Qwen recommends temperature=1.0, top_p=0.95, top_k=20 for thinking mode.

GGUF with llama.cpp. Unsloth's run command:

unsloth run --model unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL

The people in the Unsloth speed thread aren't on Blackwell workstations. One reports 19 t/s decode on an RTX 3090 with 128 GB DDR4 at Q4_K_XL. Another gets 39.48 t/s at depth 0 on an M1 Ultra 128GB at IQ4_NL, using a llama.cpp branch with Metal optimizations. On an RTX 6000 PRO, one user reports 59.70 t/s TG at Q4_K_XL, and Unsloth's docs quote 170 t/s with MTP on one RTX 6000 PRO against a 100 t/s baseline, without naming the quant. NVIDIA publishes no throughput numbers for the FP4 build at all, which is a strange omission for a release aimed at local inference.

If you own an RTX PRO 6000 or a DGX Spark, try the NVIDIA build. QAD is a real method, and a 0.7-point drop is about as good as a 4-bit-class quant gets. Everyone else should stick with GGUF, which runs in mainline llama.cpp on the hardware you already have without a single exported env var. Re-check once upstream vLLM merges the scale format, because the 55-variable setup is the part I'd least want to maintain.

So: do you trust a vendor FP4 quant over community GGUFs from unsloth and bartowski? Has anyone run FP4-Local and a Q4_K_XL side by side on the same prompts and seen a quality difference? Post your numbers, especially on image input, since NVIDIA didn't.