HP Z8 Fury G6i -- Perfect Qwen Flash Next setup (2x RTX Pro 6000s)

Be sure to check out our full review and other content featuring the HP Z8 Fury G6i

Host: 2x RTX PRO 6000 Blackwell (96 GB), 125 GB RAM, 48 cores Intel Xeon X658X

Model: Qwen/Qwen3.8-Flash-Next-FP8 (125B MoE / 6B active, 51B n-gram table, 4B MTP)

Engine: vLLM vllm/vllm-openai:qwen38-flash-next

(0.1.dev20073)

Status: SERVING on port 8000, model qwen3.8-flash-next

, 1M context, MTP enabled

*YaRN needs more testing, really, probably.

MTP acceptance: 98-100%, mean acceptance length ~4.0 (near-perfect drafting).

Here’s my startup script (I’m a little surprised this was so easy? and reachable on this system?)

#!/bin/bash
set -euo pipefail

MODE="${1:-baseline}"   # baseline (262K) or 1m
MTP="${2:-mtp}"         # mtp or nomtp

NAME="vllm-qwen38-fpn"
MODEL_DIR="/home/w/models/Qwen3.8-Flash-Next-FP8"
PORT=8000

ARGS=(
  --model /models/Qwen3.8-Flash-Next-FP8
  --tensor-parallel-size 2
  --gpu-memory-utilization 0.90
  --max-num-seqs 256
  --enable-prefix-caching
  --no-enable-flashinfer-autotune
  --enable-auto-tool-choice
  --tool-call-parser qwen3_xml
  --reasoning-parser qwen3
  --served-model-name qwen3.8-flash-next
  --port "$PORT"
)

if [ "$MODE" = "1m" ]; then
  ARGS+=(
    --max-model-len 1000000
    --gpu-memory-utilization 0.95
    --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'
  )
else
  ARGS+=(--max-model-len 262144)
fi

if [ "$MTP" = "mtp" ]; then
  ARGS+=(
    --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
  )
fi

echo "=== Launching vLLM: mode=$MODE mtp=$MTP ==="
echo "docker run -d --name $NAME ... ${ARGS[*]}"

docker rm -f "$NAME" >/dev/null 2>&1 || true

docker run -d --name "$NAME" --restart unless-stopped \
  --ipc=host --gpus all \
  --shm-size 16g \
  -e VLLM_PLE_CPU_OFFLOAD=1 \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -v "$MODEL_DIR":/models/Qwen3.8-Flash-Next-FP8:ro \
  -p "$PORT":8000 \
  vllm/vllm-openai:qwen38-flash-next \
  "${ARGS[@]}"

echo "Container $NAME started. Follow logs: docker logs -f $NAME"