rtx6000_pd.md

A developer published a guide for running disaggregated prefill-decode (P/D) inference across two nodes of NVIDIA RTX PRO 6000 (Blackwell SM120) GPUs using vLLM's NixlConnector to transfer KV cache over RDMA. The setup separates compute-heavy prefill from memory-bandwidth-heavy decode, which the guide says improves Time To First Token and cluster goodput, and requires a CUDA 13.x-compatible vLLM image plus a P/D-aware router such as NVIDIA Dynamo or llm-d for production.

This guide details how to set up a Disaggregated Prefill-Decode P/D architecture across two nodes equipped with NVIDIA RTX PRO 6000 Blackwell SM120 GPUs. By separating the compute-heavy prefill phase from the memory-bandwidth-heavy decode phase, you prevent cross-interference and significantly improve both Time To First Token TTFT and overall cluster goodput. - GPUs: 2x Nodes, each with RTX PRO 6000 GPUs 96GB GDDR7 . - Networking: High-bandwidth RDMA networking e.g., Mellanox ConnectX-6/7 is highly recommended. A 70B model with a 10k-token prompt generates ~3GB of KV cache; transferring this over standard TCP will bottleneck your TTFT. - Host Driver: CUDA 13.4+ compatible NVIDIA driver. Because the RTX PRO 6000 utilizes the Blackwell SM120 architecture, you must use a CUDA 13.x compatible vLLM image. The local-inference-lab Karmic Kraken beta or the official vLLM nightly are recommended. Recommended image for RTX PRO 6000 IMAGE="ghcr.io/local-inference-lab/vllm:karmic-kraken-beta" Alternatively: IMAGE="vllm/vllm-openai:cu130-nightly" For both nodes, ensure your Docker commands include the necessary flags for RDMA and memory locking: - --cap-add=IPC LOCK - --device /dev/infiniband if using Mellanox RDMA - --shm-size=16g The Decode worker receives the computed KV cache blocks from the Prefill node and handles the autoregressive token generation. We start this node first so it is ready to accept connections. Run on the Decode Node: Replace <PREFILL IP with the private IP of your Prefill Node export PREFILL IP="10.0.0.1" docker run --gpus all --cap-add=IPC LOCK --shm-size=16g -p 8201:8201 \ -e VLLM KV TRANSFER CONFIG='{"kv connector":"NixlConnector","kv role":"kv consumer","kv buffer device":"cuda","kv connector extra config":{"prefill addr":"'$PREFILL IP':8200"}}' \ $IMAGE python -m vllm.entrypoints.openai.api server \ --model meta-llama/Llama-3.1-70B-Instruct \ --port 8201 The Prefill worker processes the user's entire prompt, calculates the KV cache, and pushes it over the network via the NixlConnector to the Decode worker. Run on the Prefill Node: Replace <DECODE IP with the private IP of your Decode Node export DECODE IP="10.0.0.2" docker run --gpus all --cap-add=IPC LOCK --shm-size=16g -p 8200:8200 \ -e VLLM KV TRANSFER CONFIG='{"kv connector":"NixlConnector","kv role":"kv producer","kv buffer device":"cuda","kv connector extra config":{"decode addr":"'$DECODE IP':8201"}}' \ $IMAGE python -m vllm.entrypoints.openai.api server \ --model meta-llama/Llama-3.1-70B-Instruct \ --port 8200 Standard HTTP load balancers cannot handle P/D routing because the router must extract the KV cache network pointer from the prefill worker and pass it to the decode worker. For testing, you can use the toy proxy included in the vLLM repository. Run this on whichever machine will serve as your ingress gateway. Clone vllm to get the testing script if not already present git clone https://github.com/vllm-project/vllm.git cd vllm Run the proxy, pointing it to both workers python tests/disagg prefill/toy proxy server.py \ --prefill http://10.0.0.1:8200 \ --decode http://10.0.0.2:8201 \ --port 8000 Note: For production, replace toy proxy server.py with a robust P/D routing solution like NVIDIA Dynamo or llm-d . You can now send standard OpenAI-compatible requests to the proxy on port 8000 . The proxy will seamlessly split the workload, leveraging the massive VRAM and compute of your RTX PRO 6000s. curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Llama-3.1-70B-Instruct", "messages": {"role": "system", "content": "You are a helpful AI assistant."}, {"role": "user", "content": "Explain the architectural advantages of the Blackwell SM120 GPU."} , "max tokens": 1024 }'