{"slug": "agent-instructions-for-deploying-qwen-3-8-flash-next", "title": "Agent instructions for deploying Qwen 3.8 Flash Next", "summary": "A developer documented a runbook for deploying the Qwen 3.8 Flash Next model on a single NVIDIA Blackwell GPU using a specialized vLLM runtime. The deployment supports both Docker and native systemd tracks, requiring approximately 96 GB of VRAM and a 126 GiB NVFP4 checkpoint. The guide emphasizes using the specialized runtime to avoid silent fallback to stock vLLM.", "body_md": "This runbook reproduces a proven single-GPU deployment of\n`RadixArk/Qwen3.8-Flash-Next-NVFP4`\n\non an NVIDIA Blackwell workstation.\n\nThe runbook provides two final forms: a persistent Docker deployment and a native user-level systemd service. In the native track, Docker is used only once to extract the specialized vLLM Python runtime.\n\n``` php\nspecialized compatibility image\n  |-> Track A: Docker Compose -> vLLM in container\n  `-> Track B: extract dist-packages\n        -> systemd --user -> launcher -> Python 3.12 venv + PYTHONPATH\n\nEither track -> local NVFP4 checkpoint -> one Blackwell GPU\n             -> OpenAI-compatible HTTP API\n```\n\nThe virtual environment provides the Python executable. The effective vLLM,\nTorch, Transformers, CUDA Python packages, FlashInfer, and compiled extensions\ncome from the extracted `dist-packages`\n\ndirectory because it is placed first\non `PYTHONPATH`\n\n.\n\nThe verified deployment used:\n\n- Linux x86-64\n- NVIDIA Blackwell, compute capability 12.0\n- Approximately 96 GB of VRAM\n- NVIDIA driver 610-series or a compatible newer driver\n- Host CUDA toolkit 13.x\n- Python 3.12.13\n- Specialized vLLM\n`0.1.dev20073+g8e685d198`\n\n- Torch\n`2.13.0+cu130`\n\n- Transformers\n`5.15.1`\n\n- FlashInfer\n`0.6.17`\n\n- Triton\n`3.7.1`\n\n- ModelOpt NVFP4 checkpoint of approximately 126 GiB\n\nOther versions may work, but this model depends on experimental Qwen3.8 and\nModelOpt NVFP4 support. Do not replace the specialized runtime with ordinary\nPyPI vLLM unless that build demonstrably contains\n`vllm/models/qwen3_8_flash_next`\n\nand supports this checkpoint.\n\nPlan for at least:\n\n- One Blackwell GPU with about 96 GB VRAM for the single-GPU configuration\n- 150 GB for the model\n- 20 GB for the extracted runtime\n- Temporary Docker storage for an image of roughly 9 GB compressed / 29 GB as\nreported by\n`docker images`\n\n- Substantial system RAM; the checkpoint is larger than host RAM on some workstations and startup may use swap\n\nAt a 262,144-token context, the known-good server had only about one full-length request worth of KV cache. Two concurrent requests are practical when their contexts are shorter.\n\nBoth tracks use the same model and patched specialized image:\n\n| Track | Runtime | Best when | Main trade-off |\n|---|---|---|---|\n| A: Docker | Everything stays in the container | Portability and clean host isolation matter most | Docker must remain installed and running |\n| B: Extracted native | Docker packages are copied out and run with a host venv | Native systemd integration or avoiding a serving container matters most | The Python environment is an unusual two-layer setup |\n\nComplete the artifact acquisition and compatibility-image steps once, then follow either Track A or Track B. Do not do the extraction steps for Track A.\n\nBefore starting, obtain:\n\n-\nAccess to\n\n`RadixArk/Qwen3.8-Flash-Next-NVFP4`\n\n. Its repository is currently visible on Hugging Face, although its model card calls it a private candidate release. Availability or gating can change, so authenticate if required. -\nA specialized vLLM Docker image containing Qwen3.8-Flash-Next support. The known-good base image was tagged locally as:\n\n```\nvllm/vllm-openai:qwen38-flash-next\n```\n\nThe exact known-good image was locally built and records no public source revision or registry URL. The package identifies itself as commit-like build\n\n`8e685d198`\n\n, but that is not enough to claim a reproducible public checkout. Obtain it from a trusted supplier or build an explicitly Qwen3.8-capable revision as described below.\n\nThe deployment must stop rather than silently fall back to stock vLLM if either artifact is unavailable.\n\nThe standalone CLI can be installed with `uv`\n\n:\n\n```\nuv tool install huggingface_hub\nhf --help\n```\n\nIf the repository is gated, log in interactively:\n\n```\nhf auth login\n```\n\nDo not put the token on a command line, in a Dockerfile, or in source control.\n\nSet the destination and download the exact repository:\n\n```\nexport QWEN38_MODEL=\"$HOME/models/RadixArk/Qwen3.8-Flash-Next-NVFP4\"\n\nmkdir -p \"$QWEN38_MODEL\"\nhf download RadixArk/Qwen3.8-Flash-Next-NVFP4 \\\n  --local-dir \"$QWEN38_MODEL\"\n```\n\nThis checkpoint, not the roughly 360 GB BF16 source model, is the artifact used\nby this runbook. The publisher describes it as routed-expert NVFP4 with FP8 PLE\ntables and a size around 135 GB decimal / 126 GiB as reported by `du`\n\n.\n\nUse one of the following methods, in preference order.\n\nAsk the supplier for the complete registry name and digest, then pull by digest:\n\n```\ndocker pull REGISTRY/PROJECT/vllm-qwen38@sha256:SUPPLIED_DIGEST\ndocker tag REGISTRY/PROJECT/vllm-qwen38@sha256:SUPPLIED_DIGEST \\\n  vllm/vllm-openai:qwen38-flash-next\n```\n\nDo not substitute `vllm/vllm-openai:latest`\n\nmerely because the specialized tag\nis unavailable. Validate it using the checks below.\n\nOn the working machine:\n\n```\ndocker image inspect vllm/vllm-openai:qwen38-flash-next\ndocker save vllm/vllm-openai:qwen38-flash-next \\\n  | zstd -T0 -19 -o vllm-qwen38-flash-next.tar.zst\nsha256sum vllm-qwen38-flash-next.tar.zst \\\n  > vllm-qwen38-flash-next.tar.zst.sha256\n```\n\nTransfer both files using the site's approved mechanism. On the destination:\n\n```\nsha256sum --check vllm-qwen38-flash-next.tar.zst.sha256\nzstd -dc vllm-qwen38-flash-next.tar.zst | docker load\ndocker image inspect vllm/vllm-openai:qwen38-flash-next\n```\n\nThe checksum protects transfer integrity; it is not a substitute for trusting the machine or person that produced the archive.\n\nUse this only when the exact source revision has been identified and reviewed.\nThe official vLLM repository supplies `docker/Dockerfile`\n\nand a\n`vllm-openai`\n\nbuild target:\n\n```\ngit clone https://github.com/vllm-project/vllm.git\ncd vllm\ngit checkout QWEN38_CAPABLE_COMMIT_OR_TAG\n\ntest -d vllm/models/qwen3_8_flash_next\n\nDOCKER_BUILDKIT=1 docker build . \\\n  --file docker/Dockerfile \\\n  --target vllm-openai \\\n  --tag vllm/vllm-openai:qwen38-flash-next \\\n  --build-arg CUDA_VERSION=13.0.1 \\\n  --build-arg torch_cuda_arch_list='12.0' \\\n  --build-arg max_jobs=\"$(nproc)\" \\\n  --build-arg nvcc_threads=2\n```\n\nDo not literally use `QWEN38_CAPABLE_COMMIT_OR_TAG`\n\n; replace it with the\nrevision provided for the deployment. Building arbitrary current `main`\n\nmakes\nthe result difficult to audit and may change behavior.\n\nThe known-good image contains a development runtime newer and more specialized than the checkpoint's published SGLang-only serving recipe. Treat this vLLM path as a validated integration, not as the checkpoint publisher's official support promise.\n\n```\ndocker run --rm \\\n  --entrypoint python \\\n  vllm/vllm-openai:qwen38-flash-next \\\n  -c 'import torch, vllm; print(vllm.__version__, vllm.__file__); print(torch.__version__)'\n\ndocker run --rm \\\n  --entrypoint test \\\n  vllm/vllm-openai:qwen38-flash-next \\\n  -d /usr/local/lib/python3.12/dist-packages/vllm/models/qwen3_8_flash_next\n```\n\nFor the known-good artifact, the first command reports vLLM\n`0.1.dev20073+g8e685d198`\n\nand Torch `2.13.0+cu130`\n\n. If versions differ, record\nthem and perform a full smoke test rather than assuming equivalence.\n\nInstall Docker, `uv`\n\n, the NVIDIA driver, and a CUDA 13.x toolkit using the\ndistribution's normal package manager. Then verify:\n\n```\nnvidia-smi\n/opt/cuda/bin/nvcc --version\ndocker version\nuv --version\n```\n\nConfirm that `nvidia-smi`\n\nreports a Blackwell GPU with enough free VRAM and\ncompute capability 12.0.\n\nThe final native process uses the host NVIDIA driver. The driver must be new enough for the CUDA 13 runtime shipped in the extracted image.\n\nThis runbook uses user-owned paths and requires no personal username:\n\n```\nexport QWEN38_ROOT=\"$HOME/.local/opt/qwen38-vllm\"\nexport QWEN38_MODEL=\"$HOME/models/RadixArk/Qwen3.8-Flash-Next-NVFP4\"\nexport QWEN38_RUNTIME=\"$QWEN38_ROOT/runtime/dist-packages\"\nexport QWEN38_VENV=\"$QWEN38_ROOT/venv\"\n\nmkdir -p \"$QWEN38_ROOT/runtime\" \"$QWEN38_MODEL\" \"$QWEN38_ROOT/bin\"\n```\n\nThese variables are setup conveniences only. The launcher created below derives its paths independently and does not depend on an interactive shell retaining them.\n\nAfter completing the artifact-download section, validate the result:\n\n```\ntest -f \"$QWEN38_MODEL/config.json\"\ntest -f \"$QWEN38_MODEL/model.safetensors.index.json\"\ndu -sh \"$QWEN38_MODEL\"\n```\n\nExpect approximately 126 GiB. Do not start the service with a partial model download.\n\nThe checkpoint mixes NVFP4 routed experts with an FP8 PLE table. The specialized base image contains the FP8 PLE implementation, but the known-good revision selects it automatically only for globally FP8 checkpoints. Apply the explicit opt-in patch below.\n\nCreate a temporary build directory and save this as `Dockerfile`\n\n:\n\n```\nFROM vllm/vllm-openai:qwen38-flash-next\n\n# Allow the mixed NVFP4 checkpoint to explicitly select the FP8 PLE table.\nRUN sed -i '/^import math$/a import os' \\\n      /usr/local/lib/python3.12/dist-packages/vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py \\\n    && sed -i '/^    if not isinstance(quant_config, Fp8Config):$/i\\    if os.environ.get(\"VLLM_PLE_FP8_CHECKPOINT\") == \"1\":\\n        return Qwen3_8FlashNextPLEFp8EmbeddingMethod()' \\\n      /usr/local/lib/python3.12/dist-packages/vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py\n```\n\nBuild it:\n\n```\ndocker build -t vllm-qwen38-flash-next:fp8-ple-compat .\n```\n\nThe patch is intentionally tied to the known-good source layout. If the build fails because the target lines or file moved, inspect the new implementation instead of weakening or blindly changing the patch.\n\nThis track uses the compatibility image directly. The host stores the model and Docker stores the compilation cache; no host Python environment is needed.\n\nCreate an empty deployment directory, enter it, and save the following as\n`compose.yaml`\n\n:\n\n```\nservices:\n  qwen38-vllm:\n    image: vllm-qwen38-flash-next:fp8-ple-compat\n    container_name: qwen38-vllm\n    restart: unless-stopped\n    gpus: all\n    ipc: host\n    environment:\n      VLLM_PLE_CPU_OFFLOAD: \"1\"\n      VLLM_PLE_FP8_CHECKPOINT: \"1\"\n      TORCH_CUDA_ARCH_LIST: \"12.0f\"\n      PYTORCH_ALLOC_CONF: expandable_segments:True\n    ports:\n      - \"127.0.0.1:8003:8003\"\n    volumes:\n      - type: bind\n        source: ${QWEN38_MODEL:?set QWEN38_MODEL to the host model directory}\n        target: /models/Qwen3.8-Flash-Next-NVFP4\n        read_only: true\n      - type: volume\n        source: vllm-cache\n        target: /root/.cache/vllm\n    command:\n      - /models/Qwen3.8-Flash-Next-NVFP4\n      - --served-model-name\n      - Qwen/Qwen3.8-Flash-Next\n      - --host\n      - 0.0.0.0\n      - --port\n      - \"8003\"\n      - --max-model-len\n      - \"262144\"\n      - --gpu-memory-utilization\n      - \"0.96\"\n      - --tensor-parallel-size\n      - \"1\"\n      - --distributed-executor-backend\n      - mp\n      - --max-num-seqs\n      - \"2\"\n      - --max-num-batched-tokens\n      - \"8192\"\n      - --kv-cache-dtype\n      - auto\n      - --enable-prefix-caching\n      - --enable-prompt-tokens-details\n      - --no-enable-flashinfer-autotune\n      - --speculative-config\n      - '{\"method\":\"mtp\",\"num_speculative_tokens\":3}'\n      - --enable-auto-tool-choice\n      - --tool-call-parser\n      - qwen3_xml\n      - --reasoning-parser\n      - qwen3\n\nvolumes:\n  vllm-cache:\n```\n\nWhy these Docker settings matter:\n\n`gpus: all`\n\nexposes the NVIDIA GPU through NVIDIA Container Toolkit.`ipc: host`\n\nsupplies the shared-memory behavior used by PyTorch multiprocessing and the PLE offload worker.- The model is mounted read-only and is not copied into the image.\n- The named cache volume preserves Torch, Triton, and AOT compilation artifacts across container replacement.\n- The server listens on all interfaces inside the container, while the host publishes it only on loopback.\n\nThe image's entrypoint is already `vllm serve`\n\n, so `command`\n\ncontains only the\nmodel path and server arguments.\n\nFrom the Compose directory:\n\n```\nexport QWEN38_MODEL=\"$HOME/models/RadixArk/Qwen3.8-Flash-Next-NVFP4\"\n\ndocker compose config\ndocker compose up -d\ndocker compose logs -f qwen38-vllm\n```\n\nWait for `Application startup complete`\n\n, then validate:\n\n```\ncurl --fail http://127.0.0.1:8003/health\ncurl --fail http://127.0.0.1:8003/version\ncurl --fail http://127.0.0.1:8003/v1/models | jq .\nnvidia-smi\n```\n\nOperational commands:\n\n```\ndocker compose ps\ndocker compose logs --tail 200 qwen38-vllm\ndocker compose restart qwen38-vllm\ndocker compose stop\ndocker compose up -d\ndocker compose down\n```\n\n`docker compose down`\n\nremoves the container and network but retains the named\ncache volume unless `--volumes`\n\nis explicitly supplied. Do not add `--volumes`\n\nunless discarding the compilation cache is intentional.\n\nChange the port mapping only when network access is intentional:\n\n```\nports:\n  - \"8003:8003\"\n```\n\nThis runtime does not configure an API key. Use a host firewall, authenticated reverse proxy, VPN, or SSH tunnel; do not directly expose it to an untrusted network.\n\nCompose is preferred for repeatability. For a one-container deployment:\n\n```\ndocker run -d \\\n  --name qwen38-vllm \\\n  --restart unless-stopped \\\n  --gpus all \\\n  --ipc=host \\\n  -p 127.0.0.1:8003:8003 \\\n  -v \"$QWEN38_MODEL:/models/Qwen3.8-Flash-Next-NVFP4:ro\" \\\n  -v qwen38-vllm-cache:/root/.cache/vllm \\\n  -e VLLM_PLE_CPU_OFFLOAD=1 \\\n  -e VLLM_PLE_FP8_CHECKPOINT=1 \\\n  -e TORCH_CUDA_ARCH_LIST=12.0f \\\n  -e PYTORCH_ALLOC_CONF=expandable_segments:True \\\n  vllm-qwen38-flash-next:fp8-ple-compat \\\n  /models/Qwen3.8-Flash-Next-NVFP4 \\\n  --served-model-name Qwen/Qwen3.8-Flash-Next \\\n  --host 0.0.0.0 \\\n  --port 8003 \\\n  --max-model-len 262144 \\\n  --gpu-memory-utilization 0.96 \\\n  --tensor-parallel-size 1 \\\n  --distributed-executor-backend mp \\\n  --max-num-seqs 2 \\\n  --max-num-batched-tokens 8192 \\\n  --kv-cache-dtype auto \\\n  --enable-prefix-caching \\\n  --enable-prompt-tokens-details \\\n  --no-enable-flashinfer-autotune \\\n  --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser qwen3_xml \\\n  --reasoning-parser qwen3\n```\n\nFollow it with:\n\n```\ndocker logs -f qwen38-vllm\n```\n\nThis is the hybrid design of the verified workstation deployment. Docker is used to package and transfer the exact runtime, then removed from the serving path.\n\nCreate a stopped container and copy its complete Python `dist-packages`\n\ntree:\n\n```\nexport QWEN38_CONTAINER=\"qwen38-runtime-extract\"\nmkdir -p \"$QWEN38_RUNTIME\"\n\ndocker create \\\n  --name \"$QWEN38_CONTAINER\" \\\n  --entrypoint /bin/true \\\n  vllm-qwen38-flash-next:fp8-ple-compat\n\ndocker cp \\\n  \"$QWEN38_CONTAINER:/usr/local/lib/python3.12/dist-packages/.\" \\\n  \"$QWEN38_RUNTIME/\"\n\ndocker rm \"$QWEN38_CONTAINER\"\n```\n\nVerify the extracted model integration exists:\n\n```\ntest -d \"$QWEN38_RUNTIME/vllm/models/qwen3_8_flash_next\"\ntest -d \"$QWEN38_RUNTIME/nvidia_cutlass_dsl/dsl_packages\"\ndu -sh \"$QWEN38_ROOT/runtime\"\n```\n\nThe known-good extracted tree was about 16 GB.\n\nDocker is no longer used after this point. Retain the image until the native service is validated; it can be archived or removed later according to local retention policy.\n\nInstall and select Python 3.12.13:\n\n```\nuv python install 3.12.13\nuv venv --python 3.12.13 \"$QWEN38_VENV\"\n```\n\nTest the combined interpreter and extracted runtime:\n\n```\nPYTHONPATH=\"$QWEN38_RUNTIME/nvidia_cutlass_dsl/dsl_packages:$QWEN38_RUNTIME\" \\\n  \"$QWEN38_VENV/bin/python\" - <<'PY'\nimport torch\nimport vllm\n\nprint(\"vLLM:\", vllm.__version__, vllm.__file__)\nprint(\"Torch:\", torch.__version__)\nprint(\"CUDA available:\", torch.cuda.is_available())\nprint(\"GPU:\", torch.cuda.get_device_name(0))\nPY\n```\n\nConfirm that `vllm.__file__`\n\nis under the extracted runtime, not under the\nvenv's own `site-packages`\n\n. For the known-good artifact, the reported vLLM\nversion is `0.1.dev20073+g8e685d198`\n\n.\n\nSave the following as `$QWEN38_ROOT/bin/serve.sh`\n\n:\n\n``` bash\n#!/usr/bin/env bash\nset -euo pipefail\n\nROOT_DIR=\"${QWEN38_ROOT:-${HOME}/.local/opt/qwen38-vllm}\"\nMODEL_DIR=\"${MODEL_DIR:-${HOME}/models/RadixArk/Qwen3.8-Flash-Next-NVFP4}\"\nRUNTIME_DIR=\"${RUNTIME_DIR:-${ROOT_DIR}/runtime/dist-packages}\"\nPYTHON_BIN=\"${PYTHON_BIN:-${ROOT_DIR}/venv/bin/python}\"\nHOST=\"${HOST:-127.0.0.1}\"\nPORT=\"${PORT:-8003}\"\n\nif [[ ! -f \"${MODEL_DIR}/model.safetensors.index.json\" ]]; then\n  echo \"Model is incomplete or missing at ${MODEL_DIR}\" >&2\n  exit 1\nfi\n\nif [[ ! -d \"${RUNTIME_DIR}/vllm/models/qwen3_8_flash_next\" ]]; then\n  echo \"Specialized Qwen3.8 vLLM runtime is missing at ${RUNTIME_DIR}\" >&2\n  exit 1\nfi\n\nif ss -ltnH \"sport = :${PORT}\" | grep -q .; then\n  echo \"Port ${PORT} is already in use\" >&2\n  exit 1\nfi\n\nexport PYTHONPATH=\"${RUNTIME_DIR}/nvidia_cutlass_dsl/dsl_packages:${RUNTIME_DIR}\"\nexport VLLM_PLE_CPU_OFFLOAD=1\nexport VLLM_PLE_FP8_CHECKPOINT=1\nexport TORCH_CUDA_ARCH_LIST=12.0f\nexport PYTORCH_ALLOC_CONF=expandable_segments:True\n\nexec \"${PYTHON_BIN}\" -m vllm.entrypoints.cli.main serve \\\n  \"${MODEL_DIR}\" \\\n  --served-model-name Qwen/Qwen3.8-Flash-Next \\\n  --host \"${HOST}\" \\\n  --port \"${PORT}\" \\\n  --max-model-len 262144 \\\n  --gpu-memory-utilization 0.96 \\\n  --tensor-parallel-size 1 \\\n  --distributed-executor-backend mp \\\n  --max-num-seqs 2 \\\n  --max-num-batched-tokens 8192 \\\n  --kv-cache-dtype auto \\\n  --enable-prefix-caching \\\n  --enable-prompt-tokens-details \\\n  --no-enable-flashinfer-autotune \\\n  --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser qwen3_xml \\\n  --reasoning-parser qwen3\n```\n\nThen make it executable:\n\n```\nchmod 0755 \"$QWEN38_ROOT/bin/serve.sh\"\n```\n\nThe secure default binds only to `127.0.0.1`\n\n. To reproduce a LAN-visible bind,\nset `HOST=0.0.0.0`\n\nin the optional environment file described below, and put\nauthentication or a trusted reverse proxy/firewall in front of the service.\n\nCreate `$HOME/.config/systemd/user/vllm-qwen38-flash-next.service`\n\n:\n\n```\n[Unit]\nDescription=vLLM Qwen3.8-Flash-Next NVFP4 server\nAfter=network-online.target\nWants=network-online.target\n\n[Service]\nType=simple\nWorkingDirectory=%h\nEnvironmentFile=-%h/.config/vllm/qwen38.env\nExecStart=/usr/bin/env bash -lc 'exec \"$${QWEN38_ROOT:-$${HOME}/.local/opt/qwen38-vllm}/bin/serve.sh\"'\nRestart=on-failure\nRestartSec=10\nTimeoutStartSec=infinity\nTimeoutStopSec=240\n\n[Install]\nWantedBy=default.target\n```\n\nIf different paths are required, create `$HOME/.config/vllm/qwen38.env`\n\n:\n\n```\n# Values are examples. Do not put API tokens in this file.\nQWEN38_ROOT=/data/apps/qwen38-vllm\nMODEL_DIR=/data/models/RadixArk/Qwen3.8-Flash-Next-NVFP4\nHOST=127.0.0.1\nPORT=8003\n```\n\nOmit the environment file when using the default paths.\n\nEnable and start the service:\n\n```\nsystemctl --user daemon-reload\nsystemctl --user enable --now vllm-qwen38-flash-next.service\n```\n\nTo allow the user service to start during boot without an interactive login, an administrator can enable lingering for the deployment account:\n\n```\nsudo loginctl enable-linger \"$USER\"\n```\n\nModel loading and warmup can take several minutes:\n\n```\nsystemctl --user status vllm-qwen38-flash-next.service\njournalctl --user -u vllm-qwen38-flash-next.service -f\n```\n\nHealthy startup should include messages similar to:\n\n```\nResolved architecture: Qwen3_8FlashNextMTP\nDetected ModelOpt NVFP4 checkpoint\nUsing 'FLASHINFER_CUTLASS' NvFp4 MoE backend\nGPU KV cache size: approximately 266,000 tokens\nApplication startup complete\n```\n\nWarnings about experimental NVFP4 metadata, unrecognized rope keys, or Triton JIT compilation were present in the known-good deployment. Treat a worker exit, CUDA error, missing symbol, OOM, or repeated systemd restart as a failure.\n\nCheck health, the effective runtime version, and model registration:\n\n```\ncurl --fail http://127.0.0.1:8003/health\ncurl --fail http://127.0.0.1:8003/version\ncurl --fail http://127.0.0.1:8003/v1/models | jq .\n```\n\nExpected model ID:\n\n```\nQwen/Qwen3.8-Flash-Next\n```\n\nRun a minimal chat request:\n\n```\ncurl --fail http://127.0.0.1:8003/v1/chat/completions \\\n  -H 'Content-Type: application/json' \\\n  -d '{\n    \"model\": \"Qwen/Qwen3.8-Flash-Next\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Reply with: ready\"}],\n    \"max_tokens\": 16,\n    \"temperature\": 0\n  }' | jq .\n```\n\nInspect GPU residency after loading:\n\n```\nnvidia-smi\n```\n\nApproximately 94 GB of occupied VRAM is normal for the known-good configuration.\n\n```\n# Status\nsystemctl --user status vllm-qwen38-flash-next\n\n# Restart after configuration changes\nsystemctl --user restart vllm-qwen38-flash-next\n\n# Stop\nsystemctl --user stop vllm-qwen38-flash-next\n\n# Recent logs\njournalctl --user -u vllm-qwen38-flash-next -n 200 --no-pager\n\n# Live logs\njournalctl --user -u vllm-qwen38-flash-next -f\n```\n\n`127.0.0.1`\n\nis the recommended bind address when clients run locally.`0.0.0.0`\n\nexposes the API on every host interface permitted by the firewall.- The launcher does not configure an API key. Do not expose it directly to an untrusted network.\n- Prefer a firewall, authenticated reverse proxy, VPN, or SSH tunnel for remote access.\n- Never embed Hugging Face credentials or other secrets in the launcher, unit file, repository, or this runbook.\n\nPrint the effective package path:\n\n```\nPYTHONPATH=\"$QWEN38_RUNTIME/nvidia_cutlass_dsl/dsl_packages:$QWEN38_RUNTIME\" \\\n  \"$QWEN38_VENV/bin/python\" -c \\\n  'import vllm; print(vllm.__version__); print(vllm.__file__)'\n```\n\nThe path must point into `$QWEN38_RUNTIME`\n\n. If it points into the venv, correct\n`PYTHONPATH`\n\n.\n\nThe wrong Docker image was extracted. Obtain the specialized Qwen3.8 build; installing generic vLLM is not an equivalent repair.\n\nConfirm both conditions:\n\n```\ngrep -n VLLM_PLE_FP8_CHECKPOINT \\\n  \"$QWEN38_RUNTIME/vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py\"\n\nsystemctl --user show vllm-qwen38-flash-next \\\n  --property=Environment --no-pager\n```\n\nThe extracted runtime must contain the compatibility patch, and the launcher\nmust export `VLLM_PLE_FP8_CHECKPOINT=1`\n\nand `VLLM_PLE_CPU_OFFLOAD=1`\n\n.\n\nFirst ensure no unrelated compute process occupies the GPU:\n\n```\nnvidia-smi\n```\n\nThen reduce `--max-model-len`\n\nor `--gpu-memory-utilization`\n\ncautiously. The\nknown-good 262K configuration targets 96% utilization and leaves roughly\n7.7 GiB for KV cache.\n\nTriton may JIT-compile input shapes not covered by startup warmup. This occurred in the verified deployment and normally affects only the first matching shape.\n\nVerify both service enablement and user lingering:\n\n```\nsystemctl --user is-enabled vllm-qwen38-flash-next\nloginctl show-user \"$USER\" -p Linger\n```\n\n- Authorized checkpoint download is complete and approximately 126 GiB\n- GPU is Blackwell-class with approximately 96 GB free VRAM\n- Specialized image contains\n`vllm/models/qwen3_8_flash_next`\n\n- FP8 PLE compatibility patch is present\n- Track A: container imports the expected image packages\n- Track B: extracted runtime imports before venv packages\n- Effective vLLM and Torch versions match the supplied specialized image\n- Track A: Compose container is running with its restart policy\n- Track B: user systemd service is enabled and stable\n-\n`/health`\n\n,`/version`\n\n, and`/v1/models`\n\nsucceed - A chat completion succeeds\n- Bind address and network controls match the deployment's trust boundary\n- No credentials or machine-specific personal paths are stored in files\n\n- Checkpoint and publisher notes:\n[https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4](https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4) - Base-model files and upstream usage notes:\n[https://huggingface.co/Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) - Official vLLM Docker deployment and source-build instructions:\n[https://docs.vllm.ai/en/latest/deployment/docker/](https://docs.vllm.ai/en/latest/deployment/docker/) - vLLM source repository:\n[https://github.com/vllm-project/vllm](https://github.com/vllm-project/vllm)", "url": "https://wpnews.pro/news/agent-instructions-for-deploying-qwen-3-8-flash-next", "canonical_source": "https://gist.github.com/lkarlslund/c646508341c5b5c3fb9c152df194c84c", "published_at": "2026-08-28 06:02:33+00:00", "updated_at": "2026-08-30 02:52:11.396115+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["RadixArk", "Qwen3.8-Flash-Next", "NVIDIA", "vLLM", "Hugging Face", "Docker", "systemd", "ModelOpt"], "alternates": {"html": "https://wpnews.pro/news/agent-instructions-for-deploying-qwen-3-8-flash-next", "markdown": "https://wpnews.pro/news/agent-instructions-for-deploying-qwen-3-8-flash-next.md", "text": "https://wpnews.pro/news/agent-instructions-for-deploying-qwen-3-8-flash-next.txt", "jsonld": "https://wpnews.pro/news/agent-instructions-for-deploying-qwen-3-8-flash-next.jsonld"}}