Clustering Two NVIDIA DGX Sparks to Serve Qwen3-30B-Thinking with Ray + vLLM

Two NVIDIA DGX Spark units were clustered over a 200 GbE link to serve the Qwen3-30B-A3B-Thinking model using Ray and vLLM with tensor parallelism. The setup required pinning all transport layers to the high-speed NIC and using the `--reasoning-parser deepseek_r1` flag instead of the Qwen parser to avoid silent loss of reasoning traces. The configuration validates multi-node inference for large language models and provides a scalable pattern for future deployments.

Clustering Two NVIDIA DGX Sparks to Serve Qwen3-30B-Thinking with Ray + vLLM TL;DR We took two NVIDIA DGX Spark units, wired them together over a 200 GbE link, joined them into a single Ray cluster running inside a vLLM container, and serve Qwen/Qwen3-30B-A3B-Thinking-2507-FP8 with tensor parallelism across both boxes. One Spark holds shard 0, the other holds shard 1, Ray dispatches the work, and vLLM exposes an OpenAI-compatible endpoint on port 8000. The trickiest part wasn't the networking or the orchestration. It was a one-line vLLM flag — --reasoning-parser deepseek r1 — that we had to use instead of the obvious qwen3 parser, because the model emits its reasoning block without an opening <think tag and the strict Qwen parser silently swallows the whole reasoning trace. This post walks through the setup end to end and the gotcha at the end. Why two Sparks? A single DGX Spark GB10, SM121, 121 GB unified memory is plenty for any 30B-class model in FP8 — we run Qwen3.6-35B-A3B-FP8 on a single node every day. But we wanted to: - Validate the multi-node story end to end before we needed it for something bigger. - Give Qwen3-30B-Thinking room to breathe: 128k context with the KV cache distributed across two boxes leaves much more memory for in-flight requests than the same model squeezed onto one. - Have a real-world reference for the Ray + vLLM + 200 GbE pattern that we can scale to 4 or 8 Sparks later. The model itself — Qwen3-30B-A3B-Thinking — is a Mixture-of-Experts with 30B total / 3B active params and an explicit "thinking" mode that emits a <think …</think reasoning trace before the answer. The hardware path: 200 GbE between the Sparks Each Spark has two QSFP cages on its ConnectX NIC, presented to Linux as enp1s0f0np0 and enp1s0f1np1 . We cabled enp1s0f1np1 on Node 1 directly to the same port on Node 2 — no switch in the middle — and brought the link up with static IPs on a small /30 . To verify which interface is the live one, use ibdev2netdev : js $ ibdev2netdev mlx5 0 port 1 == enp1s0f0np0 Down mlx5 1 port 1 == enp1s0f1np1 Up That Up line is what the head-node script reads. The whole Ray + NCCL + UCX stack pins itself to this interface so no traffic ever leaks onto the management NIC. Starting the Ray head node On Node 1, we run /usr/local/bin/run headnode.sh . It's intentionally tiny — the real plumbing is in the upstream run cluster.sh helper from the vLLM repo; the wrapper just resolves the IP and forwards the right env vars: /usr/local/bin/run headnode.sh export MN IF NAME=enp1s0f1np1 export VLLM HOST IP=$ ip -4 addr show $MN IF NAME \ | grep -oP ' ?<=inet\s \d+ \.\d+ {3}' export VLLM IMAGE=nvcr.io/nvidia/vllm:25.11-py3 echo "Using interface $MN IF NAME with IP $VLLM HOST IP" bash run cluster.sh $VLLM IMAGE $VLLM HOST IP --head ~/.cache/huggingface \ -e VLLM HOST IP=$VLLM HOST IP \ -e UCX NET DEVICES=$MN IF NAME \ -e NCCL SOCKET IFNAME=$MN IF NAME \ -e OMPI MCA btl tcp if include=$MN IF NAME \ -e GLOO SOCKET IFNAME=$MN IF NAME \ -e TP SOCKET IFNAME=$MN IF NAME \ -e RAY memory monitor refresh ms=0 \ -e MASTER ADDR=$VLLM HOST IP A few things worth pointing out: Every transport gets pinned to the same NIC. UCX, NCCL, OMPI, Gloo, and vLLM's TP socket all read separate env vars. Setting only one of them is the classic mistake — the other libraries happily fall back to eth0 and you get a healthy-looking cluster that runs at 1 GbE speeds. Setting all five means every byte goes over the 200 GbE link.so the model weights are pre-staged on disk run ~/.cache/huggingface is bind-mounted hf download Qwen/Qwen3-30B-A3B-Thinking-2507-FP8 once before you start and shared between the host and the container.disables Ray's OOM killer. On Spark's unified memory, Ray's heuristic is too aggressive and kills the vLLM worker before it's finished allocating KV cache. RAY memory monitor refresh ms=0 - The container itself runs nvcr.io/nvidia/vllm:25.11-py3 . NGC's vLLM image already has the right CUDA / NCCL / FlashInfer stack for GB10 — building your own from PyPI is a world of pain we don't recommend. When this script finishes you have a container named node-0 or node-1 depending on the cluster helper's counter running on Node 1, with Ray's head process listening on port 6379. Starting the worker On Node 2 we have the symmetric script. It's the same run cluster.sh invocation, but --head is replaced with --worker , and the second positional arg is the head node's IP, not its own: /usr/local/bin/run workernode.sh on Node 2 On Node 2, join as worker Set the interface name same as Node 1 export MN IF NAME=enp1s0f1np1 Get Node 2's own IP address export VLLM HOST IP=$ ip -4 addr show $MN IF NAME | grep -oP ' ?<=inet\s \d+ \.\d+ {3}' IMPORTANT: Set HEAD NODE IP to Node 1's IP address export HEAD NODE IP=10.0.0.3 Set vLLM image export VLLM IMAGE=nvcr.io/nvidia/vllm:25.11-py3 echo "Worker IP: $VLLM HOST IP, connecting to head node at: $HEAD NODE IP" bash run cluster.sh $VLLM IMAGE $HEAD NODE IP --worker ~/.cache/huggingface \ -e VLLM HOST IP=$VLLM HOST IP \ -e UCX NET DEVICES=$MN IF NAME \ -e NCCL SOCKET IFNAME=$MN IF NAME \ -e OMPI MCA btl tcp if include=$MN IF NAME \ -e GLOO SOCKET IFNAME=$MN IF NAME \ -e TP SOCKET IFNAME=$MN IF NAME \ -e RAY memory monitor refresh ms=0 \ -e MASTER ADDR=$HEAD NODE IP The same interface-pinning rules apply on the worker. Two things to notice: is the head node's address on the 200 GbE HEAD NODE IP=10.0.0.3 /30 we put on enp1s0f1np1 — not its management IP. If you point this at the wrong NIC the workers will still find Ray, but every NCCL collective will quietly fall back to the slow path.here, not local. Ray uses it to elect the rank-0 coordinator for the vLLM TP group; getting this wrong on the worker is a common reason tensor-parallel init hangs forever at startup. MASTER ADDR=$HEAD NODE IP To confirm the two nodes have actually joined, exec into either container and run ray status . We have that wrapped in /usr/local/bin/ray inference health.sh : export VLLM CONTAINER=$ docker ps --format '{{.Names}}' | grep -E '^node- 0-9 +$' docker exec $VLLM CONTAINER ray status curl http://127.0.0.1:8000/health docker exec $VLLM CONTAINER nvidia-smi --query-gpu=memory.used,memory.total --format=csv You should see two nodes, two GPUs total, and one resource type called GPU with value 2 . If you see only one, the worker isn't reaching the head — usually because something on the network path is firewalled or the wrong interface got picked up. Launching vLLM inside the Ray cluster With Ray up and both nodes joined, the actual model launch is a docker exec into the head container. That's ~/docker/vllm/qwen36/launch.sh : bash /usr/bin/env bash set -euo pipefail SCRIPT DIR="$ cd "$ dirname "${BASH SOURCE 0 }" " && pwd " if -f "${SCRIPT DIR}/.env" ; then set -a; source "${SCRIPT DIR}/.env"; set +a fi : "${VLLM API KEY:?VLLM API KEY not set expected in .env }" VLLM CONTAINER=$ docker ps --format '{{.Names}}' | grep -E '^node- 0-9 +$' | head -n1 if -z "${VLLM CONTAINER}" ; then echo "No node- container running. Start the Ray cluster first." &2 exit 1 fi docker exec -it -e VLLM API KEY="${VLLM API KEY}" "${VLLM CONTAINER}" /bin/bash -c ' set -e exec vllm serve Qwen/Qwen3-30B-A3B-Thinking-2507-FP8 \ --served-model-name qwen3 30b thinking \ --host 0.0.0.0 --port 8000 \ --tensor-parallel-size 2 \ --max-num-seqs 8 \ --max-model-len 131072 \ --max-num-batched-tokens 32768 \ --gpu-memory-utilization 0.70 \ --enable-prefix-caching \ --reasoning-parser deepseek r1 \ --enable-auto-tool-choice \ --tool-call-parser hermes ' Key decisions: — Ray sees two GPUs across two nodes and shards one half of the model to each. vLLM doesn't need --tensor-parallel-size 2 --pipeline-parallel-size ; TP across two nodes is what we want for a 30B MoE.— the model supports 256k via YaRN scaling, but 128k is what we actually need and it leaves comfortable KV headroom. --max-model-len 131072 — conservative on purpose. Spark's unified memory layout means GPU allocations and the host can fight; 70% reliably avoids OOM under load. --gpu-memory-utilization 0.70 — large win for system-prompted workloads, which is most of ours. --enable-prefix-caching — Qwen3 thinking models emit tool calls in the Hermes format. Auto-choice lets the model decide when to call a tool vs. answer directly. --enable-auto-tool-choice --tool-call-parser hermes Secrets: .env next to launch.sh launch.sh sources ~/docker/vllm/qwen36/.env with set -a so every variable is exported into the environment. The file contains exactly two values: ~/docker/vllm/qwen36/.env mode 0600, .gitignored HF TOKEN=hf xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx VLLM API KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx HF TOKEN is needed at first run so the container can download the FP8 weights after the first run they live in ~/.cache/huggingface and the token isn't reached for again . VLLM API KEY is the bearer token clients must send as Authorization: Bearer … . vLLM picks it up automatically when the env var is set inside the serving process — that's why launch.sh passes it through with docker exec -e VLLM API KEY=… . The : "${VLLM API KEY:?…}" guard fails the script fast if the .env is missing or empty, instead of starting an open server. We keep this file chmod 600 , never check it into git, and rotate both secrets when anyone leaves the team. The gotcha: <think and the wrong reasoning parser Here's the one that cost us an afternoon. Qwen3-30B-Thinking emits its reasoning trace inside <think …</think tags. vLLM ships a parser called qwen3 that's purpose-built for this format. The obvious thing to do is: --reasoning-parser qwen3 Don't. With this model and the FP8 image we run, the output stream looks like this: js …this is a multi-step problem, let me think about prime factorisation… …</think The answer is 42. Notice what's not there: an opening <think tag. The model's chat template adds it as the last token of the prompt , so generation begins already inside the thinking block. The first thing the model ever emits is the reasoning text itself — the opening tag is implicit. The qwen3 parser is strict and waits for an opening tag that never arrives. The whole reasoning trace ends up in the wrong field of the OpenAI response or gets dropped entirely, depending on the vLLM version , and clients see an empty reasoning content followed by content that starts with </think . The fix is to use DeepSeek-R1's parser instead: --reasoning-parser deepseek r1 deepseek r1 is lenient about the opening tag — it treats everything from the start of the response up to a closing </think as reasoning, and everything after as the answer. That happens to be exactly the shape Qwen3-Thinking produces in practice. Reasoning lands in choices 0 .message.reasoning content , the answer lands in choices 0 .message.content , and tool-calls round-trip correctly through the Hermes parser. If you ever see truncated or missing reasoning content with a Qwen thinking model, this is almost certainly the cause. The strict-vs-lenient mismatch isn't documented prominently in either project, but it's load-bearing once you wire the model into a real client. What we ended up with - Two DGX Sparks, one 200 GbE cable between them. - Ray cluster running inside nvcr.io/nvidia/vllm:25.11-py3 containers, networking pinned to the high-speed NIC across UCX / NCCL / OMPI / Gloo / vLLM-TP. vllm serve Qwen/Qwen3-30B-A3B-Thinking-2507-FP8 with TP=2, 128k context, prefix caching, Hermes tool calls.- OpenAI-compatible endpoint on port 8000 with a real bearer token. - Inference at multi-node throughput, with the reasoning trace surfacing cleanly thanks to one carefully chosen parser flag. Next stop: scaling the same recipe to four Sparks for a 70B-class model. The wrapper scripts barely change — just --tensor-parallel-size 4 and one more --worker invocation per box. Appendix: useful one-liners Health check curl -s -H "Authorization: Bearer $VLLM API KEY" http://localhost:8000/v1/models | jq . Ray topology from inside the container docker exec $ docker ps --format '{{.Names}}' | grep -E '^node- 0-9 +$' ray status Watch GPU memory on both Sparks run on each watch -n 1 'nvidia-smi --query-gpu=memory.used,memory.total --format=csv' Tail the vLLM serving process docker logs -f $ docker ps --format '{{.Names}}' | grep -E '^node- 0-9 +$' | head -n1