rtx6000_pd.md

This guide details how to set up a Disaggregated Prefill-Decode (P/D) architecture across two nodes equipped with NVIDIA RTX PRO 6000 (Blackwell SM120) GPUs.

By separating the compute-heavy prefill phase from the memory-bandwidth-heavy decode phase, you prevent cross-interference and significantly improve both Time To First Token (TTFT) and overall cluster goodput.

• GPUs: 2x Nodes, each with RTX PRO 6000 GPUs (96GB GDDR7).
• Networking: High-bandwidth RDMA networking (e.g., Mellanox ConnectX-6/7) is highly recommended. A 70B model with a 10k-token prompt generates ~3GB of KV cache; transferring this over standard TCP will bottleneck your TTFT.
• Host Driver: CUDA 13.4+ compatible NVIDIA driver.

Because the RTX PRO 6000 utilizes the Blackwell SM120 architecture, you must use a CUDA 13.x compatible vLLM image. The local-inference-lab Karmic Kraken beta or the official vLLM nightly are recommended.

IMAGE="ghcr.io/local-inference-lab/vllm:karmic-kraken-beta"

For both nodes, ensure your Docker commands include the necessary flags for RDMA and memory locking:

• --cap-add=IPC_LOCK
• --device /dev/infiniband (if using Mellanox RDMA)
• --shm-size=16g

The Decode worker receives the computed KV cache blocks from the Prefill node and handles the autoregressive token generation. We start this node first so it is ready to accept connections.

Run on the Decode Node:

export PREFILL_IP="10.0.0.1" 

docker run --gpus all --cap-add=IPC_LOCK --shm-size=16g -p 8201:8201 \
  -e VLLM_KV_TRANSFER_CONFIG='{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_buffer_device":"cuda","kv_connector_extra_config":{"prefill_addr":"'$PREFILL_IP':8200"}}' \
  $IMAGE python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --port 8201

The Prefill worker processes the user's entire prompt, calculates the KV cache, and pushes it over the network via the NixlConnector to the Decode worker.

Run on the Prefill Node:

export DECODE_IP="10.0.0.2"

docker run --gpus all --cap-add=IPC_LOCK --shm-size=16g -p 8200:8200 \
  -e VLLM_KV_TRANSFER_CONFIG='{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_buffer_device":"cuda","kv_connector_extra_config":{"decode_addr":"'$DECODE_IP':8201"}}' \
  $IMAGE python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --port 8200

Standard HTTP load balancers cannot handle P/D routing because the router must extract the KV cache network pointer from the prefill worker and pass it to the decode worker.

For testing, you can use the toy proxy included in the vLLM repository. Run this on whichever machine will serve as your ingress gateway.

git clone https://github.com/vllm-project/vllm.git
cd vllm

python tests/disagg_prefill/toy_proxy_server.py \
  --prefill http://10.0.0.1:8200 \
  --decode http://10.0.0.2:8201 \
  --port 8000

Note: For production, replace toy_proxy_server.py with a robust P/D routing solution like NVIDIA Dynamo or llm-d.

You can now send standard OpenAI-compatible requests to the proxy on port 8000. The proxy will seamlessly split the workload, leveraging the massive VRAM and compute of your RTX PRO 6000s.

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-70B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a helpful AI assistant."},
      {"role": "user", "content": "Explain the architectural advantages of the Blackwell SM120 GPU."}
    ],
    "max_tokens": 1024
  }'