This runbook reproduces a proven single-GPU deployment of
RadixArk/Qwen3.8-Flash-Next-NVFP4
on an NVIDIA Blackwell workstation.
The runbook provides two final forms: a persistent Docker deployment and a native user-level systemd service. In the native track, Docker is used only once to extract the specialized vLLM Python runtime.
specialized compatibility image
|-> Track A: Docker Compose -> vLLM in container
`-> Track B: extract dist-packages
-> systemd --user -> launcher -> Python 3.12 venv + PYTHONPATH
Either track -> local NVFP4 checkpoint -> one Blackwell GPU
-> OpenAI-compatible HTTP API
The virtual environment provides the Python executable. The effective vLLM,
Torch, Transformers, CUDA Python packages, FlashInfer, and compiled extensions
come from the extracted dist-packages
directory because it is placed first
on PYTHONPATH
.
The verified deployment used:
-
Linux x86-64
-
NVIDIA Blackwell, compute capability 12.0
-
Approximately 96 GB of VRAM
-
NVIDIA driver 610-series or a compatible newer driver
-
Host CUDA toolkit 13.x
-
Python 3.12.13
-
Specialized vLLM
0.1.dev20073+g8e685d198 -
Torch
2.13.0+cu130 -
Transformers
5.15.1 -
FlashInfer
0.6.17 -
Triton
3.7.1 -
ModelOpt NVFP4 checkpoint of approximately 126 GiB
Other versions may work, but this model depends on experimental Qwen3.8 and
ModelOpt NVFP4 support. Do not replace the specialized runtime with ordinary
PyPI vLLM unless that build demonstrably contains
vllm/models/qwen3_8_flash_next
and supports this checkpoint.
Plan for at least:
-
One Blackwell GPU with about 96 GB VRAM for the single-GPU configuration
-
150 GB for the model
-
20 GB for the extracted runtime
-
Temporary Docker storage for an image of roughly 9 GB compressed / 29 GB as reported by
docker images -
Substantial system RAM; the checkpoint is larger than host RAM on some workstations and startup may use swap
At a 262,144-token context, the known-good server had only about one full-length request worth of KV cache. Two concurrent requests are practical when their contexts are shorter.
Both tracks use the same model and patched specialized image:
| Track | Runtime | Best when | Main trade-off |
|---|---|---|---|
| A: Docker | Everything stays in the container | Portability and clean host isolation matter most | Docker must remain installed and running |
| B: Extracted native | Docker packages are copied out and run with a host venv | Native systemd integration or avoiding a serving container matters most | The Python environment is an unusual two-layer setup |
Complete the artifact acquisition and compatibility-image steps once, then follow either Track A or Track B. Do not do the extraction steps for Track A.
Before starting, obtain:
Access to
RadixArk/Qwen3.8-Flash-Next-NVFP4
. Its repository is currently visible on Hugging Face, although its model card calls it a private candidate release. Availability or gating can change, so authenticate if required. - A specialized vLLM Docker image containing Qwen3.8-Flash-Next support. The known-good base image was tagged locally as:
vllm/vllm-openai:qwen38-flash-next
The exact known-good image was locally built and records no public source revision or registry URL. The package identifies itself as commit-like build
8e685d198
, but that is not enough to claim a reproducible public checkout. Obtain it from a trusted supplier or build an explicitly Qwen3.8-capable revision as described below.
The deployment must stop rather than silently fall back to stock vLLM if either artifact is unavailable.
The standalone CLI can be installed with uv
:
uv tool install huggingface_hub
hf --help
If the repository is gated, log in interactively:
hf auth login
Do not put the token on a command line, in a Dockerfile, or in source control.
Set the destination and download the exact repository:
export QWEN38_MODEL="$HOME/models/RadixArk/Qwen3.8-Flash-Next-NVFP4"
mkdir -p "$QWEN38_MODEL"
hf download RadixArk/Qwen3.8-Flash-Next-NVFP4 \
--local-dir "$QWEN38_MODEL"
This checkpoint, not the roughly 360 GB BF16 source model, is the artifact used
by this runbook. The publisher describes it as routed-expert NVFP4 with FP8 PLE
tables and a size around 135 GB decimal / 126 GiB as reported by du
.
Use one of the following methods, in preference order.
Ask the supplier for the complete registry name and digest, then pull by digest:
docker pull REGISTRY/PROJECT/vllm-qwen38@sha256:SUPPLIED_DIGEST
docker tag REGISTRY/PROJECT/vllm-qwen38@sha256:SUPPLIED_DIGEST \
vllm/vllm-openai:qwen38-flash-next
Do not substitute vllm/vllm-openai:latest
merely because the specialized tag is unavailable. Validate it using the checks below.
On the working machine:
docker image inspect vllm/vllm-openai:qwen38-flash-next
docker save vllm/vllm-openai:qwen38-flash-next \
| zstd -T0 -19 -o vllm-qwen38-flash-next.tar.zst
sha256sum vllm-qwen38-flash-next.tar.zst \
> vllm-qwen38-flash-next.tar.zst.sha256
Transfer both files using the site's approved mechanism. On the destination:
sha256sum --check vllm-qwen38-flash-next.tar.zst.sha256
zstd -dc vllm-qwen38-flash-next.tar.zst | docker load
docker image inspect vllm/vllm-openai:qwen38-flash-next
The checksum protects transfer integrity; it is not a substitute for trusting the machine or person that produced the archive.
Use this only when the exact source revision has been identified and reviewed.
The official vLLM repository supplies docker/Dockerfile
and a
vllm-openai
build target:
git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout QWEN38_CAPABLE_COMMIT_OR_TAG
test -d vllm/models/qwen3_8_flash_next
DOCKER_BUILDKIT=1 docker build . \
--file docker/Dockerfile \
--target vllm-openai \
--tag vllm/vllm-openai:qwen38-flash-next \
--build-arg CUDA_VERSION=13.0.1 \
--build-arg torch_cuda_arch_list='12.0' \
--build-arg max_jobs="$(nproc)" \
--build-arg nvcc_threads=2
Do not literally use QWEN38_CAPABLE_COMMIT_OR_TAG
; replace it with the
revision provided for the deployment. Building arbitrary current main
makes the result difficult to audit and may change behavior.
The known-good image contains a development runtime newer and more specialized than the checkpoint's published SGLang-only serving recipe. Treat this vLLM path as a validated integration, not as the checkpoint publisher's official support promise.
docker run --rm \
--entrypoint python \
vllm/vllm-openai:qwen38-flash-next \
-c 'import torch, vllm; print(vllm.__version__, vllm.__file__); print(torch.__version__)'
docker run --rm \
--entrypoint test \
vllm/vllm-openai:qwen38-flash-next \
-d /usr/local/lib/python3.12/dist-packages/vllm/models/qwen3_8_flash_next
For the known-good artifact, the first command reports vLLM
0.1.dev20073+g8e685d198
and Torch 2.13.0+cu130
. If versions differ, record them and perform a full smoke test rather than assuming equivalence.
Install Docker, uv
, the NVIDIA driver, and a CUDA 13.x toolkit using the distribution's normal package manager. Then verify:
nvidia-smi
/opt/cuda/bin/nvcc --version
docker version
uv --version
Confirm that nvidia-smi
reports a Blackwell GPU with enough free VRAM and compute capability 12.0.
The final native process uses the host NVIDIA driver. The driver must be new enough for the CUDA 13 runtime shipped in the extracted image.
This runbook uses user-owned paths and requires no personal username:
export QWEN38_ROOT="$HOME/.local/opt/qwen38-vllm"
export QWEN38_MODEL="$HOME/models/RadixArk/Qwen3.8-Flash-Next-NVFP4"
export QWEN38_RUNTIME="$QWEN38_ROOT/runtime/dist-packages"
export QWEN38_VENV="$QWEN38_ROOT/venv"
mkdir -p "$QWEN38_ROOT/runtime" "$QWEN38_MODEL" "$QWEN38_ROOT/bin"
These variables are setup conveniences only. The launcher created below derives its paths independently and does not depend on an interactive shell retaining them.
After completing the artifact-download section, validate the result:
test -f "$QWEN38_MODEL/config.json"
test -f "$QWEN38_MODEL/model.safetensors.index.json"
du -sh "$QWEN38_MODEL"
Expect approximately 126 GiB. Do not start the service with a partial model download.
The checkpoint mixes NVFP4 routed experts with an FP8 PLE table. The specialized base image contains the FP8 PLE implementation, but the known-good revision selects it automatically only for globally FP8 checkpoints. Apply the explicit opt-in patch below.
Create a temporary build directory and save this as Dockerfile
:
FROM vllm/vllm-openai:qwen38-flash-next
RUN sed -i '/^import math$/a import os' \
/usr/local/lib/python3.12/dist-packages/vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py \
&& sed -i '/^ if not isinstance(quant_config, Fp8Config):$/i\ if os.environ.get("VLLM_PLE_FP8_CHECKPOINT") == "1":\n return Qwen3_8FlashNextPLEFp8EmbeddingMethod()' \
/usr/local/lib/python3.12/dist-packages/vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py
Build it:
docker build -t vllm-qwen38-flash-next:fp8-ple-compat .
The patch is intentionally tied to the known-good source layout. If the build fails because the target lines or file moved, inspect the new implementation instead of weakening or blindly changing the patch.
This track uses the compatibility image directly. The host stores the model and Docker stores the compilation cache; no host Python environment is needed.
Create an empty deployment directory, enter it, and save the following as
compose.yaml
:
services:
qwen38-vllm:
image: vllm-qwen38-flash-next:fp8-ple-compat
container_name: qwen38-vllm
restart: unless-stopped
gpus: all
ipc: host
environment:
VLLM_PLE_CPU_OFFLOAD: "1"
VLLM_PLE_FP8_CHECKPOINT: "1"
TORCH_CUDA_ARCH_LIST: "12.0f"
PYTORCH_ALLOC_CONF: expandable_segments:True
ports:
- "127.0.0.1:8003:8003"
volumes:
- type: bind
source: ${QWEN38_MODEL:?set QWEN38_MODEL to the host model directory}
target: /models/Qwen3.8-Flash-Next-NVFP4
read_only: true
- type: volume
source: vllm-cache
target: /root/.cache/vllm
command:
- /models/Qwen3.8-Flash-Next-NVFP4
- --served-model-name
- Qwen/Qwen3.8-Flash-Next
- --host
- 0.0.0.0
- --port
- "8003"
- --max-model-len
- "262144"
- --gpu-memory-utilization
- "0.96"
- --tensor-parallel-size
- "1"
- --distributed-executor-backend
- mp
- --max-num-seqs
- "2"
- --max-num-batched-tokens
- "8192"
- --kv-cache-dtype
- auto
- --enable-prefix-caching
- --enable-prompt-tokens-details
- --no-enable-flashinfer-autotune
- --speculative-config
- '{"method":"mtp","num_speculative_tokens":3}'
- --enable-auto-tool-choice
- --tool-call-parser
- qwen3_xml
- --reasoning-parser
- qwen3
volumes:
vllm-cache:
Why these Docker settings matter:
gpus: all
exposes the NVIDIA GPU through NVIDIA Container Toolkit.ipc: host
supplies the shared-memory behavior used by PyTorch multiprocessing and the PLE offload worker.- The model is mounted read-only and is not copied into the image.
- The named cache volume preserves Torch, Triton, and AOT compilation artifacts across container replacement.
- The server listens on all interfaces inside the container, while the host publishes it only on loopback.
The image's entrypoint is already vllm serve
, so command
contains only the model path and server arguments.
From the Compose directory:
export QWEN38_MODEL="$HOME/models/RadixArk/Qwen3.8-Flash-Next-NVFP4"
docker compose config
docker compose up -d
docker compose logs -f qwen38-vllm
Wait for Application startup complete
, then validate:
curl --fail http://127.0.0.1:8003/health
curl --fail http://127.0.0.1:8003/version
curl --fail http://127.0.0.1:8003/v1/models | jq .
nvidia-smi
Operational commands:
docker compose ps
docker compose logs --tail 200 qwen38-vllm
docker compose restart qwen38-vllm
docker compose stop
docker compose up -d
docker compose down
docker compose down
removes the container and network but retains the named
cache volume unless --volumes
is explicitly supplied. Do not add --volumes
unless discarding the compilation cache is intentional.
Change the port mapping only when network access is intentional:
ports:
- "8003:8003"
This runtime does not configure an API key. Use a host firewall, authenticated reverse proxy, VPN, or SSH tunnel; do not directly expose it to an untrusted network.
Compose is preferred for repeatability. For a one-container deployment:
docker run -d \
--name qwen38-vllm \
--restart unless-stopped \
--gpus all \
--ipc=host \
-p 127.0.0.1:8003:8003 \
-v "$QWEN38_MODEL:/models/Qwen3.8-Flash-Next-NVFP4:ro" \
-v qwen38-vllm-cache:/root/.cache/vllm \
-e VLLM_PLE_CPU_OFFLOAD=1 \
-e VLLM_PLE_FP8_CHECKPOINT=1 \
-e TORCH_CUDA_ARCH_LIST=12.0f \
-e PYTORCH_ALLOC_CONF=expandable_segments:True \
vllm-qwen38-flash-next:fp8-ple-compat \
/models/Qwen3.8-Flash-Next-NVFP4 \
--served-model-name Qwen/Qwen3.8-Flash-Next \
--host 0.0.0.0 \
--port 8003 \
--max-model-len 262144 \
--gpu-memory-utilization 0.96 \
--tensor-parallel-size 1 \
--distributed-executor-backend mp \
--max-num-seqs 2 \
--max-num-batched-tokens 8192 \
--kv-cache-dtype auto \
--enable-prefix-caching \
--enable-prompt-tokens-details \
--no-enable-flashinfer-autotune \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--reasoning-parser qwen3
Follow it with:
docker logs -f qwen38-vllm
This is the hybrid design of the verified workstation deployment. Docker is used to package and transfer the exact runtime, then removed from the serving path.
Create a stopped container and copy its complete Python dist-packages
tree:
export QWEN38_CONTAINER="qwen38-runtime-extract"
mkdir -p "$QWEN38_RUNTIME"
docker create \
--name "$QWEN38_CONTAINER" \
--entrypoint /bin/true \
vllm-qwen38-flash-next:fp8-ple-compat
docker cp \
"$QWEN38_CONTAINER:/usr/local/lib/python3.12/dist-packages/." \
"$QWEN38_RUNTIME/"
docker rm "$QWEN38_CONTAINER"
Verify the extracted model integration exists:
test -d "$QWEN38_RUNTIME/vllm/models/qwen3_8_flash_next"
test -d "$QWEN38_RUNTIME/nvidia_cutlass_dsl/dsl_packages"
du -sh "$QWEN38_ROOT/runtime"
The known-good extracted tree was about 16 GB.
Docker is no longer used after this point. Retain the image until the native service is validated; it can be archived or removed later according to local retention policy.
Install and select Python 3.12.13:
uv python install 3.12.13
uv venv --python 3.12.13 "$QWEN38_VENV"
Test the combined interpreter and extracted runtime:
PYTHONPATH="$QWEN38_RUNTIME/nvidia_cutlass_dsl/dsl_packages:$QWEN38_RUNTIME" \
"$QWEN38_VENV/bin/python" - <<'PY'
import torch
import vllm
print("vLLM:", vllm.__version__, vllm.__file__)
print("Torch:", torch.__version__)
print("CUDA available:", torch.cuda.is_available())
print("GPU:", torch.cuda.get_device_name(0))
PY
Confirm that vllm.__file__
is under the extracted runtime, not under the
venv's own site-packages
. For the known-good artifact, the reported vLLM
version is 0.1.dev20073+g8e685d198
.
Save the following as $QWEN38_ROOT/bin/serve.sh
:
#!/usr/bin/env bash
set -euo pipefail
ROOT_DIR="${QWEN38_ROOT:-${HOME}/.local/opt/qwen38-vllm}"
MODEL_DIR="${MODEL_DIR:-${HOME}/models/RadixArk/Qwen3.8-Flash-Next-NVFP4}"
RUNTIME_DIR="${RUNTIME_DIR:-${ROOT_DIR}/runtime/dist-packages}"
PYTHON_BIN="${PYTHON_BIN:-${ROOT_DIR}/venv/bin/python}"
HOST="${HOST:-127.0.0.1}"
PORT="${PORT:-8003}"
if [[ ! -f "${MODEL_DIR}/model.safetensors.index.json" ]]; then
echo "Model is incomplete or missing at ${MODEL_DIR}" >&2
exit 1
fi
if [[ ! -d "${RUNTIME_DIR}/vllm/models/qwen3_8_flash_next" ]]; then
echo "Specialized Qwen3.8 vLLM runtime is missing at ${RUNTIME_DIR}" >&2
exit 1
fi
if ss -ltnH "sport = :${PORT}" | grep -q .; then
echo "Port ${PORT} is already in use" >&2
exit 1
fi
export PYTHONPATH="${RUNTIME_DIR}/nvidia_cutlass_dsl/dsl_packages:${RUNTIME_DIR}"
export VLLM_PLE_CPU_OFFLOAD=1
export VLLM_PLE_FP8_CHECKPOINT=1
export TORCH_CUDA_ARCH_LIST=12.0f
export PYTORCH_ALLOC_CONF=expandable_segments:True
exec "${PYTHON_BIN}" -m vllm.entrypoints.cli.main serve \
"${MODEL_DIR}" \
--served-model-name Qwen/Qwen3.8-Flash-Next \
--host "${HOST}" \
--port "${PORT}" \
--max-model-len 262144 \
--gpu-memory-utilization 0.96 \
--tensor-parallel-size 1 \
--distributed-executor-backend mp \
--max-num-seqs 2 \
--max-num-batched-tokens 8192 \
--kv-cache-dtype auto \
--enable-prefix-caching \
--enable-prompt-tokens-details \
--no-enable-flashinfer-autotune \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--reasoning-parser qwen3
Then make it executable:
chmod 0755 "$QWEN38_ROOT/bin/serve.sh"
The secure default binds only to 127.0.0.1
. To reproduce a LAN-visible bind,
set HOST=0.0.0.0
in the optional environment file described below, and put authentication or a trusted reverse proxy/firewall in front of the service.
Create $HOME/.config/systemd/user/vllm-qwen38-flash-next.service
:
[Unit]
Description=vLLM Qwen3.8-Flash-Next NVFP4 server
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
WorkingDirectory=%h
EnvironmentFile=-%h/.config/vllm/qwen38.env
ExecStart=/usr/bin/env bash -lc 'exec "$${QWEN38_ROOT:-$${HOME}/.local/opt/qwen38-vllm}/bin/serve.sh"'
Restart=on-failure
RestartSec=10
TimeoutStartSec=infinity
TimeoutStopSec=240
[Install]
WantedBy=default.target
If different paths are required, create $HOME/.config/vllm/qwen38.env
:
QWEN38_ROOT=/data/apps/qwen38-vllm
MODEL_DIR=/data/models/RadixArk/Qwen3.8-Flash-Next-NVFP4
HOST=127.0.0.1
PORT=8003
Omit the environment file when using the default paths.
Enable and start the service:
systemctl --user daemon-reload
systemctl --user enable --now vllm-qwen38-flash-next.service
To allow the user service to start during boot without an interactive login, an administrator can enable lingering for the deployment account:
sudo loginctl enable-linger "$USER"
Model and warmup can take several minutes:
systemctl --user status vllm-qwen38-flash-next.service
journalctl --user -u vllm-qwen38-flash-next.service -f
Healthy startup should include messages similar to:
Resolved architecture: Qwen3_8FlashNextMTP
Detected ModelOpt NVFP4 checkpoint
Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend
GPU KV cache size: approximately 266,000 tokens
Application startup complete
Warnings about experimental NVFP4 metadata, unrecognized rope keys, or Triton JIT compilation were present in the known-good deployment. Treat a worker exit, CUDA error, missing symbol, OOM, or repeated systemd restart as a failure.
Check health, the effective runtime version, and model registration:
curl --fail http://127.0.0.1:8003/health
curl --fail http://127.0.0.1:8003/version
curl --fail http://127.0.0.1:8003/v1/models | jq .
Expected model ID:
Qwen/Qwen3.8-Flash-Next
Run a minimal chat request:
curl --fail http://127.0.0.1:8003/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "Qwen/Qwen3.8-Flash-Next",
"messages": [{"role": "user", "content": "Reply with: ready"}],
"max_tokens": 16,
"temperature": 0
}' | jq .
Inspect GPU residency after :
nvidia-smi
Approximately 94 GB of occupied VRAM is normal for the known-good configuration.
systemctl --user status vllm-qwen38-flash-next
systemctl --user restart vllm-qwen38-flash-next
systemctl --user stop vllm-qwen38-flash-next
journalctl --user -u vllm-qwen38-flash-next -n 200 --no-pager
journalctl --user -u vllm-qwen38-flash-next -f
127.0.0.1
is the recommended bind address when clients run locally.0.0.0.0
exposes the API on every host interface permitted by the firewall.- The launcher does not configure an API key. Do not expose it directly to an untrusted network.
- Prefer a firewall, authenticated reverse proxy, VPN, or SSH tunnel for remote access.
- Never embed Hugging Face credentials or other secrets in the launcher, unit file, repository, or this runbook.
Print the effective package path:
PYTHONPATH="$QWEN38_RUNTIME/nvidia_cutlass_dsl/dsl_packages:$QWEN38_RUNTIME" \
"$QWEN38_VENV/bin/python" -c \
'import vllm; print(vllm.__version__); print(vllm.__file__)'
The path must point into $QWEN38_RUNTIME
. If it points into the venv, correct
PYTHONPATH
.
The wrong Docker image was extracted. Obtain the specialized Qwen3.8 build; installing generic vLLM is not an equivalent repair.
Confirm both conditions:
grep -n VLLM_PLE_FP8_CHECKPOINT \
"$QWEN38_RUNTIME/vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py"
systemctl --user show vllm-qwen38-flash-next \
--property=Environment --no-pager
The extracted runtime must contain the compatibility patch, and the launcher
must export VLLM_PLE_FP8_CHECKPOINT=1
and VLLM_PLE_CPU_OFFLOAD=1
.
First ensure no unrelated compute process occupies the GPU:
nvidia-smi
Then reduce --max-model-len
or --gpu-memory-utilization
cautiously. The known-good 262K configuration targets 96% utilization and leaves roughly 7.7 GiB for KV cache.
Triton may JIT-compile input shapes not covered by startup warmup. This occurred in the verified deployment and normally affects only the first matching shape.
Verify both service enablement and user lingering:
systemctl --user is-enabled vllm-qwen38-flash-next
loginctl show-user "$USER" -p Linger
-
Authorized checkpoint download is complete and approximately 126 GiB
-
GPU is Blackwell-class with approximately 96 GB free VRAM
-
Specialized image contains
vllm/models/qwen3_8_flash_next -
FP8 PLE compatibility patch is present
-
Track A: container imports the expected image packages
-
Track B: extracted runtime imports before venv packages
-
Effective vLLM and Torch versions match the supplied specialized image
-
Track A: Compose container is running with its restart policy
-
Track B: user systemd service is enabled and stable
/health
,/version
, and/v1/models
succeed - A chat completion succeeds
-
Bind address and network controls match the deployment's trust boundary
-
No credentials or machine-specific personal paths are stored in files
-
Checkpoint and publisher notes: https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4 - Base-model files and upstream usage notes: https://huggingface.co/Qwen/Qwen3.8-Flash-Next - Official vLLM Docker deployment and source-build instructions: https://docs.vllm.ai/en/latest/deployment/docker/ - vLLM source repository: https://github.com/vllm-project/vllm