# rtx6000_pd.md

> Source: <https://gist.github.com/tsg-/7148a5da0b21333b62a98f41b486ff6d>
> Published: 2026-11-10 21:56:49+00:00

This guide details how to set up a Disaggregated Prefill-Decode (P/D) architecture across two nodes equipped with NVIDIA RTX PRO 6000 (Blackwell SM120) GPUs.

By separating the compute-heavy prefill phase from the memory-bandwidth-heavy decode phase, you prevent cross-interference and significantly improve both Time To First Token (TTFT) and overall cluster goodput.

- **GPUs:** 2x Nodes, each with RTX PRO 6000 GPUs (96GB GDDR7).
- **Networking:** High-bandwidth RDMA networking (e.g., Mellanox ConnectX-6/7) is highly recommended. A 70B model with a 10k-token prompt generates ~3GB of KV cache; transferring this over standard TCP will bottleneck your TTFT.
- **Host Driver:** CUDA 13.4+ compatible NVIDIA driver.

Because the RTX PRO 6000 utilizes the Blackwell SM120 architecture, you must use a CUDA 13.x compatible vLLM image. The `local-inference-lab` Karmic Kraken beta or the official vLLM nightly are recommended.

```
# Recommended image for RTX PRO 6000
IMAGE="ghcr.io/local-inference-lab/vllm:karmic-kraken-beta"
# Alternatively: IMAGE="vllm/vllm-openai:cu130-nightly"
```

For both nodes, ensure your Docker commands include the necessary flags for RDMA and memory locking:

- `--cap-add=IPC_LOCK`
- `--device /dev/infiniband` (if using Mellanox RDMA)
- `--shm-size=16g`

The Decode worker receives the computed KV cache blocks from the Prefill node and handles the autoregressive token generation. We start this node first so it is ready to accept connections.

**Run on the Decode Node:**

```
# Replace <PREFILL_IP> with the private IP of your Prefill Node
export PREFILL_IP="10.0.0.1" 

docker run --gpus all --cap-add=IPC_LOCK --shm-size=16g -p 8201:8201 \
  -e VLLM_KV_TRANSFER_CONFIG='{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_buffer_device":"cuda","kv_connector_extra_config":{"prefill_addr":"'$PREFILL_IP':8200"}}' \
  $IMAGE python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --port 8201
```

The Prefill worker processes the user's entire prompt, calculates the KV cache, and pushes it over the network via the `NixlConnector` to the Decode worker.

**Run on the Prefill Node:**

```
# Replace <DECODE_IP> with the private IP of your Decode Node
export DECODE_IP="10.0.0.2"

docker run --gpus all --cap-add=IPC_LOCK --shm-size=16g -p 8200:8200 \
  -e VLLM_KV_TRANSFER_CONFIG='{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_buffer_device":"cuda","kv_connector_extra_config":{"decode_addr":"'$DECODE_IP':8201"}}' \
  $IMAGE python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --port 8200
```

Standard HTTP load balancers cannot handle P/D routing because the router must extract the KV cache network pointer from the prefill worker and pass it to the decode worker.

For testing, you can use the toy proxy included in the vLLM repository. Run this on whichever machine will serve as your ingress gateway.

```
# Clone vllm to get the testing script (if not already present)
git clone https://github.com/vllm-project/vllm.git
cd vllm

# Run the proxy, pointing it to both workers
python tests/disagg_prefill/toy_proxy_server.py \
  --prefill http://10.0.0.1:8200 \
  --decode http://10.0.0.2:8201 \
  --port 8000
```

*Note: For production, replace `toy_proxy_server.py` with a robust P/D routing solution like NVIDIA Dynamo or `llm-d`.*

You can now send standard OpenAI-compatible requests to the proxy on port `8000`. The proxy will seamlessly split the workload, leveraging the massive VRAM and compute of your RTX PRO 6000s.

```
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-70B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a helpful AI assistant."},
      {"role": "user", "content": "Explain the architectural advantages of the Blackwell SM120 GPU."}
    ],
    "max_tokens": 1024
  }'
```