Agent instructions for deploying Qwen 3.8 Flash Next

A developer documented a runbook for deploying the Qwen 3.8 Flash Next model on a single NVIDIA Blackwell GPU using a specialized vLLM runtime. The deployment supports both Docker and native systemd tracks, requiring approximately 96 GB of VRAM and a 126 GiB NVFP4 checkpoint. The guide emphasizes using the specialized runtime to avoid silent fallback to stock vLLM.

This runbook reproduces a proven single-GPU deployment of RadixArk/Qwen3.8-Flash-Next-NVFP4 on an NVIDIA Blackwell workstation. The runbook provides two final forms: a persistent Docker deployment and a native user-level systemd service. In the native track, Docker is used only once to extract the specialized vLLM Python runtime. php specialized compatibility image |- Track A: Docker Compose - vLLM in container - Track B: extract dist-packages - systemd --user - launcher - Python 3.12 venv + PYTHONPATH Either track - local NVFP4 checkpoint - one Blackwell GPU - OpenAI-compatible HTTP API The virtual environment provides the Python executable. The effective vLLM, Torch, Transformers, CUDA Python packages, FlashInfer, and compiled extensions come from the extracted dist-packages directory because it is placed first on PYTHONPATH . The verified deployment used: - Linux x86-64 - NVIDIA Blackwell, compute capability 12.0 - Approximately 96 GB of VRAM - NVIDIA driver 610-series or a compatible newer driver - Host CUDA toolkit 13.x - Python 3.12.13 - Specialized vLLM 0.1.dev20073+g8e685d198 - Torch 2.13.0+cu130 - Transformers 5.15.1 - FlashInfer 0.6.17 - Triton 3.7.1 - ModelOpt NVFP4 checkpoint of approximately 126 GiB Other versions may work, but this model depends on experimental Qwen3.8 and ModelOpt NVFP4 support. Do not replace the specialized runtime with ordinary PyPI vLLM unless that build demonstrably contains vllm/models/qwen3 8 flash next and supports this checkpoint. Plan for at least: - One Blackwell GPU with about 96 GB VRAM for the single-GPU configuration - 150 GB for the model - 20 GB for the extracted runtime - Temporary Docker storage for an image of roughly 9 GB compressed / 29 GB as reported by docker images - Substantial system RAM; the checkpoint is larger than host RAM on some workstations and startup may use swap At a 262,144-token context, the known-good server had only about one full-length request worth of KV cache. Two concurrent requests are practical when their contexts are shorter. Both tracks use the same model and patched specialized image: | Track | Runtime | Best when | Main trade-off | |---|---|---|---| | A: Docker | Everything stays in the container | Portability and clean host isolation matter most | Docker must remain installed and running | | B: Extracted native | Docker packages are copied out and run with a host venv | Native systemd integration or avoiding a serving container matters most | The Python environment is an unusual two-layer setup | Complete the artifact acquisition and compatibility-image steps once, then follow either Track A or Track B. Do not do the extraction steps for Track A. Before starting, obtain: - Access to RadixArk/Qwen3.8-Flash-Next-NVFP4 . Its repository is currently visible on Hugging Face, although its model card calls it a private candidate release. Availability or gating can change, so authenticate if required. - A specialized vLLM Docker image containing Qwen3.8-Flash-Next support. The known-good base image was tagged locally as: vllm/vllm-openai:qwen38-flash-next The exact known-good image was locally built and records no public source revision or registry URL. The package identifies itself as commit-like build 8e685d198 , but that is not enough to claim a reproducible public checkout. Obtain it from a trusted supplier or build an explicitly Qwen3.8-capable revision as described below. The deployment must stop rather than silently fall back to stock vLLM if either artifact is unavailable. The standalone CLI can be installed with uv : uv tool install huggingface hub hf --help If the repository is gated, log in interactively: hf auth login Do not put the token on a command line, in a Dockerfile, or in source control. Set the destination and download the exact repository: export QWEN38 MODEL="$HOME/models/RadixArk/Qwen3.8-Flash-Next-NVFP4" mkdir -p "$QWEN38 MODEL" hf download RadixArk/Qwen3.8-Flash-Next-NVFP4 \ --local-dir "$QWEN38 MODEL" This checkpoint, not the roughly 360 GB BF16 source model, is the artifact used by this runbook. The publisher describes it as routed-expert NVFP4 with FP8 PLE tables and a size around 135 GB decimal / 126 GiB as reported by du . Use one of the following methods, in preference order. Ask the supplier for the complete registry name and digest, then pull by digest: docker pull REGISTRY/PROJECT/vllm-qwen38@sha256:SUPPLIED DIGEST docker tag REGISTRY/PROJECT/vllm-qwen38@sha256:SUPPLIED DIGEST \ vllm/vllm-openai:qwen38-flash-next Do not substitute vllm/vllm-openai:latest merely because the specialized tag is unavailable. Validate it using the checks below. On the working machine: docker image inspect vllm/vllm-openai:qwen38-flash-next docker save vllm/vllm-openai:qwen38-flash-next \ | zstd -T0 -19 -o vllm-qwen38-flash-next.tar.zst sha256sum vllm-qwen38-flash-next.tar.zst \ vllm-qwen38-flash-next.tar.zst.sha256 Transfer both files using the site's approved mechanism. On the destination: sha256sum --check vllm-qwen38-flash-next.tar.zst.sha256 zstd -dc vllm-qwen38-flash-next.tar.zst | docker load docker image inspect vllm/vllm-openai:qwen38-flash-next The checksum protects transfer integrity; it is not a substitute for trusting the machine or person that produced the archive. Use this only when the exact source revision has been identified and reviewed. The official vLLM repository supplies docker/Dockerfile and a vllm-openai build target: git clone https://github.com/vllm-project/vllm.git cd vllm git checkout QWEN38 CAPABLE COMMIT OR TAG test -d vllm/models/qwen3 8 flash next DOCKER BUILDKIT=1 docker build . \ --file docker/Dockerfile \ --target vllm-openai \ --tag vllm/vllm-openai:qwen38-flash-next \ --build-arg CUDA VERSION=13.0.1 \ --build-arg torch cuda arch list='12.0' \ --build-arg max jobs="$ nproc " \ --build-arg nvcc threads=2 Do not literally use QWEN38 CAPABLE COMMIT OR TAG ; replace it with the revision provided for the deployment. Building arbitrary current main makes the result difficult to audit and may change behavior. The known-good image contains a development runtime newer and more specialized than the checkpoint's published SGLang-only serving recipe. Treat this vLLM path as a validated integration, not as the checkpoint publisher's official support promise. docker run --rm \ --entrypoint python \ vllm/vllm-openai:qwen38-flash-next \ -c 'import torch, vllm; print vllm. version , vllm. file ; print torch. version ' docker run --rm \ --entrypoint test \ vllm/vllm-openai:qwen38-flash-next \ -d /usr/local/lib/python3.12/dist-packages/vllm/models/qwen3 8 flash next For the known-good artifact, the first command reports vLLM 0.1.dev20073+g8e685d198 and Torch 2.13.0+cu130 . If versions differ, record them and perform a full smoke test rather than assuming equivalence. Install Docker, uv , the NVIDIA driver, and a CUDA 13.x toolkit using the distribution's normal package manager. Then verify: nvidia-smi /opt/cuda/bin/nvcc --version docker version uv --version Confirm that nvidia-smi reports a Blackwell GPU with enough free VRAM and compute capability 12.0. The final native process uses the host NVIDIA driver. The driver must be new enough for the CUDA 13 runtime shipped in the extracted image. This runbook uses user-owned paths and requires no personal username: export QWEN38 ROOT="$HOME/.local/opt/qwen38-vllm" export QWEN38 MODEL="$HOME/models/RadixArk/Qwen3.8-Flash-Next-NVFP4" export QWEN38 RUNTIME="$QWEN38 ROOT/runtime/dist-packages" export QWEN38 VENV="$QWEN38 ROOT/venv" mkdir -p "$QWEN38 ROOT/runtime" "$QWEN38 MODEL" "$QWEN38 ROOT/bin" These variables are setup conveniences only. The launcher created below derives its paths independently and does not depend on an interactive shell retaining them. After completing the artifact-download section, validate the result: test -f "$QWEN38 MODEL/config.json" test -f "$QWEN38 MODEL/model.safetensors.index.json" du -sh "$QWEN38 MODEL" Expect approximately 126 GiB. Do not start the service with a partial model download. The checkpoint mixes NVFP4 routed experts with an FP8 PLE table. The specialized base image contains the FP8 PLE implementation, but the known-good revision selects it automatically only for globally FP8 checkpoints. Apply the explicit opt-in patch below. Create a temporary build directory and save this as Dockerfile : FROM vllm/vllm-openai:qwen38-flash-next Allow the mixed NVFP4 checkpoint to explicitly select the FP8 PLE table. RUN sed -i '/^import math$/a import os' \ /usr/local/lib/python3.12/dist-packages/vllm/models/qwen3 8 flash next/nvidia/ple layer.py \ && sed -i '/^ if not isinstance quant config, Fp8Config :$/i\ if os.environ.get "VLLM PLE FP8 CHECKPOINT" == "1":\n return Qwen3 8FlashNextPLEFp8EmbeddingMethod ' \ /usr/local/lib/python3.12/dist-packages/vllm/models/qwen3 8 flash next/nvidia/ple layer.py Build it: docker build -t vllm-qwen38-flash-next:fp8-ple-compat . The patch is intentionally tied to the known-good source layout. If the build fails because the target lines or file moved, inspect the new implementation instead of weakening or blindly changing the patch. This track uses the compatibility image directly. The host stores the model and Docker stores the compilation cache; no host Python environment is needed. Create an empty deployment directory, enter it, and save the following as compose.yaml : services: qwen38-vllm: image: vllm-qwen38-flash-next:fp8-ple-compat container name: qwen38-vllm restart: unless-stopped gpus: all ipc: host environment: VLLM PLE CPU OFFLOAD: "1" VLLM PLE FP8 CHECKPOINT: "1" TORCH CUDA ARCH LIST: "12.0f" PYTORCH ALLOC CONF: expandable segments:True ports: - "127.0.0.1:8003:8003" volumes: - type: bind source: ${QWEN38 MODEL:?set QWEN38 MODEL to the host model directory} target: /models/Qwen3.8-Flash-Next-NVFP4 read only: true - type: volume source: vllm-cache target: /root/.cache/vllm command: - /models/Qwen3.8-Flash-Next-NVFP4 - --served-model-name - Qwen/Qwen3.8-Flash-Next - --host - 0.0.0.0 - --port - "8003" - --max-model-len - "262144" - --gpu-memory-utilization - "0.96" - --tensor-parallel-size - "1" - --distributed-executor-backend - mp - --max-num-seqs - "2" - --max-num-batched-tokens - "8192" - --kv-cache-dtype - auto - --enable-prefix-caching - --enable-prompt-tokens-details - --no-enable-flashinfer-autotune - --speculative-config - '{"method":"mtp","num speculative tokens":3}' - --enable-auto-tool-choice - --tool-call-parser - qwen3 xml - --reasoning-parser - qwen3 volumes: vllm-cache: Why these Docker settings matter: gpus: all exposes the NVIDIA GPU through NVIDIA Container Toolkit. ipc: host supplies the shared-memory behavior used by PyTorch multiprocessing and the PLE offload worker.- The model is mounted read-only and is not copied into the image. - The named cache volume preserves Torch, Triton, and AOT compilation artifacts across container replacement. - The server listens on all interfaces inside the container, while the host publishes it only on loopback. The image's entrypoint is already vllm serve , so command contains only the model path and server arguments. From the Compose directory: export QWEN38 MODEL="$HOME/models/RadixArk/Qwen3.8-Flash-Next-NVFP4" docker compose config docker compose up -d docker compose logs -f qwen38-vllm Wait for Application startup complete , then validate: curl --fail http://127.0.0.1:8003/health curl --fail http://127.0.0.1:8003/version curl --fail http://127.0.0.1:8003/v1/models | jq . nvidia-smi Operational commands: docker compose ps docker compose logs --tail 200 qwen38-vllm docker compose restart qwen38-vllm docker compose stop docker compose up -d docker compose down docker compose down removes the container and network but retains the named cache volume unless --volumes is explicitly supplied. Do not add --volumes unless discarding the compilation cache is intentional. Change the port mapping only when network access is intentional: ports: - "8003:8003" This runtime does not configure an API key. Use a host firewall, authenticated reverse proxy, VPN, or SSH tunnel; do not directly expose it to an untrusted network. Compose is preferred for repeatability. For a one-container deployment: docker run -d \ --name qwen38-vllm \ --restart unless-stopped \ --gpus all \ --ipc=host \ -p 127.0.0.1:8003:8003 \ -v "$QWEN38 MODEL:/models/Qwen3.8-Flash-Next-NVFP4:ro" \ -v qwen38-vllm-cache:/root/.cache/vllm \ -e VLLM PLE CPU OFFLOAD=1 \ -e VLLM PLE FP8 CHECKPOINT=1 \ -e TORCH CUDA ARCH LIST=12.0f \ -e PYTORCH ALLOC CONF=expandable segments:True \ vllm-qwen38-flash-next:fp8-ple-compat \ /models/Qwen3.8-Flash-Next-NVFP4 \ --served-model-name Qwen/Qwen3.8-Flash-Next \ --host 0.0.0.0 \ --port 8003 \ --max-model-len 262144 \ --gpu-memory-utilization 0.96 \ --tensor-parallel-size 1 \ --distributed-executor-backend mp \ --max-num-seqs 2 \ --max-num-batched-tokens 8192 \ --kv-cache-dtype auto \ --enable-prefix-caching \ --enable-prompt-tokens-details \ --no-enable-flashinfer-autotune \ --speculative-config '{"method":"mtp","num speculative tokens":3}' \ --enable-auto-tool-choice \ --tool-call-parser qwen3 xml \ --reasoning-parser qwen3 Follow it with: docker logs -f qwen38-vllm This is the hybrid design of the verified workstation deployment. Docker is used to package and transfer the exact runtime, then removed from the serving path. Create a stopped container and copy its complete Python dist-packages tree: export QWEN38 CONTAINER="qwen38-runtime-extract" mkdir -p "$QWEN38 RUNTIME" docker create \ --name "$QWEN38 CONTAINER" \ --entrypoint /bin/true \ vllm-qwen38-flash-next:fp8-ple-compat docker cp \ "$QWEN38 CONTAINER:/usr/local/lib/python3.12/dist-packages/." \ "$QWEN38 RUNTIME/" docker rm "$QWEN38 CONTAINER" Verify the extracted model integration exists: test -d "$QWEN38 RUNTIME/vllm/models/qwen3 8 flash next" test -d "$QWEN38 RUNTIME/nvidia cutlass dsl/dsl packages" du -sh "$QWEN38 ROOT/runtime" The known-good extracted tree was about 16 GB. Docker is no longer used after this point. Retain the image until the native service is validated; it can be archived or removed later according to local retention policy. Install and select Python 3.12.13: uv python install 3.12.13 uv venv --python 3.12.13 "$QWEN38 VENV" Test the combined interpreter and extracted runtime: PYTHONPATH="$QWEN38 RUNTIME/nvidia cutlass dsl/dsl packages:$QWEN38 RUNTIME" \ "$QWEN38 VENV/bin/python" - <<'PY' import torch import vllm print "vLLM:", vllm. version , vllm. file print "Torch:", torch. version print "CUDA available:", torch.cuda.is available print "GPU:", torch.cuda.get device name 0 PY Confirm that vllm. file is under the extracted runtime, not under the venv's own site-packages . For the known-good artifact, the reported vLLM version is 0.1.dev20073+g8e685d198 . Save the following as $QWEN38 ROOT/bin/serve.sh : bash /usr/bin/env bash set -euo pipefail ROOT DIR="${QWEN38 ROOT:-${HOME}/.local/opt/qwen38-vllm}" MODEL DIR="${MODEL DIR:-${HOME}/models/RadixArk/Qwen3.8-Flash-Next-NVFP4}" RUNTIME DIR="${RUNTIME DIR:-${ROOT DIR}/runtime/dist-packages}" PYTHON BIN="${PYTHON BIN:-${ROOT DIR}/venv/bin/python}" HOST="${HOST:-127.0.0.1}" PORT="${PORT:-8003}" if -f "${MODEL DIR}/model.safetensors.index.json" ; then echo "Model is incomplete or missing at ${MODEL DIR}" &2 exit 1 fi if -d "${RUNTIME DIR}/vllm/models/qwen3 8 flash next" ; then echo "Specialized Qwen3.8 vLLM runtime is missing at ${RUNTIME DIR}" &2 exit 1 fi if ss -ltnH "sport = :${PORT}" | grep -q .; then echo "Port ${PORT} is already in use" &2 exit 1 fi export PYTHONPATH="${RUNTIME DIR}/nvidia cutlass dsl/dsl packages:${RUNTIME DIR}" export VLLM PLE CPU OFFLOAD=1 export VLLM PLE FP8 CHECKPOINT=1 export TORCH CUDA ARCH LIST=12.0f export PYTORCH ALLOC CONF=expandable segments:True exec "${PYTHON BIN}" -m vllm.entrypoints.cli.main serve \ "${MODEL DIR}" \ --served-model-name Qwen/Qwen3.8-Flash-Next \ --host "${HOST}" \ --port "${PORT}" \ --max-model-len 262144 \ --gpu-memory-utilization 0.96 \ --tensor-parallel-size 1 \ --distributed-executor-backend mp \ --max-num-seqs 2 \ --max-num-batched-tokens 8192 \ --kv-cache-dtype auto \ --enable-prefix-caching \ --enable-prompt-tokens-details \ --no-enable-flashinfer-autotune \ --speculative-config '{"method":"mtp","num speculative tokens":3}' \ --enable-auto-tool-choice \ --tool-call-parser qwen3 xml \ --reasoning-parser qwen3 Then make it executable: chmod 0755 "$QWEN38 ROOT/bin/serve.sh" The secure default binds only to 127.0.0.1 . To reproduce a LAN-visible bind, set HOST=0.0.0.0 in the optional environment file described below, and put authentication or a trusted reverse proxy/firewall in front of the service. Create $HOME/.config/systemd/user/vllm-qwen38-flash-next.service : Unit Description=vLLM Qwen3.8-Flash-Next NVFP4 server After=network-online.target Wants=network-online.target Service Type=simple WorkingDirectory=%h EnvironmentFile=-%h/.config/vllm/qwen38.env ExecStart=/usr/bin/env bash -lc 'exec "$${QWEN38 ROOT:-$${HOME}/.local/opt/qwen38-vllm}/bin/serve.sh"' Restart=on-failure RestartSec=10 TimeoutStartSec=infinity TimeoutStopSec=240 Install WantedBy=default.target If different paths are required, create $HOME/.config/vllm/qwen38.env : Values are examples. Do not put API tokens in this file. QWEN38 ROOT=/data/apps/qwen38-vllm MODEL DIR=/data/models/RadixArk/Qwen3.8-Flash-Next-NVFP4 HOST=127.0.0.1 PORT=8003 Omit the environment file when using the default paths. Enable and start the service: systemctl --user daemon-reload systemctl --user enable --now vllm-qwen38-flash-next.service To allow the user service to start during boot without an interactive login, an administrator can enable lingering for the deployment account: sudo loginctl enable-linger "$USER" Model loading and warmup can take several minutes: systemctl --user status vllm-qwen38-flash-next.service journalctl --user -u vllm-qwen38-flash-next.service -f Healthy startup should include messages similar to: Resolved architecture: Qwen3 8FlashNextMTP Detected ModelOpt NVFP4 checkpoint Using 'FLASHINFER CUTLASS' NvFp4 MoE backend GPU KV cache size: approximately 266,000 tokens Application startup complete Warnings about experimental NVFP4 metadata, unrecognized rope keys, or Triton JIT compilation were present in the known-good deployment. Treat a worker exit, CUDA error, missing symbol, OOM, or repeated systemd restart as a failure. Check health, the effective runtime version, and model registration: curl --fail http://127.0.0.1:8003/health curl --fail http://127.0.0.1:8003/version curl --fail http://127.0.0.1:8003/v1/models | jq . Expected model ID: Qwen/Qwen3.8-Flash-Next Run a minimal chat request: curl --fail http://127.0.0.1:8003/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "Qwen/Qwen3.8-Flash-Next", "messages": {"role": "user", "content": "Reply with: ready"} , "max tokens": 16, "temperature": 0 }' | jq . Inspect GPU residency after loading: nvidia-smi Approximately 94 GB of occupied VRAM is normal for the known-good configuration. Status systemctl --user status vllm-qwen38-flash-next Restart after configuration changes systemctl --user restart vllm-qwen38-flash-next Stop systemctl --user stop vllm-qwen38-flash-next Recent logs journalctl --user -u vllm-qwen38-flash-next -n 200 --no-pager Live logs journalctl --user -u vllm-qwen38-flash-next -f 127.0.0.1 is the recommended bind address when clients run locally. 0.0.0.0 exposes the API on every host interface permitted by the firewall.- The launcher does not configure an API key. Do not expose it directly to an untrusted network. - Prefer a firewall, authenticated reverse proxy, VPN, or SSH tunnel for remote access. - Never embed Hugging Face credentials or other secrets in the launcher, unit file, repository, or this runbook. Print the effective package path: PYTHONPATH="$QWEN38 RUNTIME/nvidia cutlass dsl/dsl packages:$QWEN38 RUNTIME" \ "$QWEN38 VENV/bin/python" -c \ 'import vllm; print vllm. version ; print vllm. file ' The path must point into $QWEN38 RUNTIME . If it points into the venv, correct PYTHONPATH . The wrong Docker image was extracted. Obtain the specialized Qwen3.8 build; installing generic vLLM is not an equivalent repair. Confirm both conditions: grep -n VLLM PLE FP8 CHECKPOINT \ "$QWEN38 RUNTIME/vllm/models/qwen3 8 flash next/nvidia/ple layer.py" systemctl --user show vllm-qwen38-flash-next \ --property=Environment --no-pager The extracted runtime must contain the compatibility patch, and the launcher must export VLLM PLE FP8 CHECKPOINT=1 and VLLM PLE CPU OFFLOAD=1 . First ensure no unrelated compute process occupies the GPU: nvidia-smi Then reduce --max-model-len or --gpu-memory-utilization cautiously. The known-good 262K configuration targets 96% utilization and leaves roughly 7.7 GiB for KV cache. Triton may JIT-compile input shapes not covered by startup warmup. This occurred in the verified deployment and normally affects only the first matching shape. Verify both service enablement and user lingering: systemctl --user is-enabled vllm-qwen38-flash-next loginctl show-user "$USER" -p Linger - Authorized checkpoint download is complete and approximately 126 GiB - GPU is Blackwell-class with approximately 96 GB free VRAM - Specialized image contains vllm/models/qwen3 8 flash next - FP8 PLE compatibility patch is present - Track A: container imports the expected image packages - Track B: extracted runtime imports before venv packages - Effective vLLM and Torch versions match the supplied specialized image - Track A: Compose container is running with its restart policy - Track B: user systemd service is enabled and stable - /health , /version , and /v1/models succeed - A chat completion succeeds - Bind address and network controls match the deployment's trust boundary - No credentials or machine-specific personal paths are stored in files - Checkpoint and publisher notes: https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4 https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4 - Base-model files and upstream usage notes: https://huggingface.co/Qwen/Qwen3.8-Flash-Next https://huggingface.co/Qwen/Qwen3.8-Flash-Next - Official vLLM Docker deployment and source-build instructions: https://docs.vllm.ai/en/latest/deployment/docker/ https://docs.vllm.ai/en/latest/deployment/docker/ - vLLM source repository: https://github.com/vllm-project/vllm https://github.com/vllm-project/vllm