{"slug": "rtx6000-pd-md", "title": "rtx6000_pd.md", "summary": "A developer published a guide for running disaggregated prefill-decode (P/D) inference across two nodes of NVIDIA RTX PRO 6000 (Blackwell SM120) GPUs using vLLM's NixlConnector to transfer KV cache over RDMA. The setup separates compute-heavy prefill from memory-bandwidth-heavy decode, which the guide says improves Time To First Token and cluster goodput, and requires a CUDA 13.x-compatible vLLM image plus a P/D-aware router such as NVIDIA Dynamo or llm-d for production.", "body_md": "This guide details how to set up a Disaggregated Prefill-Decode (P/D) architecture across two nodes equipped with NVIDIA RTX PRO 6000 (Blackwell SM120) GPUs.\n\nBy separating the compute-heavy prefill phase from the memory-bandwidth-heavy decode phase, you prevent cross-interference and significantly improve both Time To First Token (TTFT) and overall cluster goodput.\n\n- **GPUs:** 2x Nodes, each with RTX PRO 6000 GPUs (96GB GDDR7).\n- **Networking:** High-bandwidth RDMA networking (e.g., Mellanox ConnectX-6/7) is highly recommended. A 70B model with a 10k-token prompt generates ~3GB of KV cache; transferring this over standard TCP will bottleneck your TTFT.\n- **Host Driver:** CUDA 13.4+ compatible NVIDIA driver.\n\nBecause the RTX PRO 6000 utilizes the Blackwell SM120 architecture, you must use a CUDA 13.x compatible vLLM image. The `local-inference-lab` Karmic Kraken beta or the official vLLM nightly are recommended.\n\n```\n# Recommended image for RTX PRO 6000\nIMAGE=\"ghcr.io/local-inference-lab/vllm:karmic-kraken-beta\"\n# Alternatively: IMAGE=\"vllm/vllm-openai:cu130-nightly\"\n```\n\nFor both nodes, ensure your Docker commands include the necessary flags for RDMA and memory locking:\n\n- `--cap-add=IPC_LOCK`\n- `--device /dev/infiniband` (if using Mellanox RDMA)\n- `--shm-size=16g`\n\nThe Decode worker receives the computed KV cache blocks from the Prefill node and handles the autoregressive token generation. We start this node first so it is ready to accept connections.\n\n**Run on the Decode Node:**\n\n```\n# Replace <PREFILL_IP> with the private IP of your Prefill Node\nexport PREFILL_IP=\"10.0.0.1\" \n\ndocker run --gpus all --cap-add=IPC_LOCK --shm-size=16g -p 8201:8201 \\\n  -e VLLM_KV_TRANSFER_CONFIG='{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_consumer\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"prefill_addr\":\"'$PREFILL_IP':8200\"}}' \\\n  $IMAGE python -m vllm.entrypoints.openai.api_server \\\n  --model meta-llama/Llama-3.1-70B-Instruct \\\n  --port 8201\n```\n\nThe Prefill worker processes the user's entire prompt, calculates the KV cache, and pushes it over the network via the `NixlConnector` to the Decode worker.\n\n**Run on the Prefill Node:**\n\n```\n# Replace <DECODE_IP> with the private IP of your Decode Node\nexport DECODE_IP=\"10.0.0.2\"\n\ndocker run --gpus all --cap-add=IPC_LOCK --shm-size=16g -p 8200:8200 \\\n  -e VLLM_KV_TRANSFER_CONFIG='{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_producer\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"decode_addr\":\"'$DECODE_IP':8201\"}}' \\\n  $IMAGE python -m vllm.entrypoints.openai.api_server \\\n  --model meta-llama/Llama-3.1-70B-Instruct \\\n  --port 8200\n```\n\nStandard HTTP load balancers cannot handle P/D routing because the router must extract the KV cache network pointer from the prefill worker and pass it to the decode worker.\n\nFor testing, you can use the toy proxy included in the vLLM repository. Run this on whichever machine will serve as your ingress gateway.\n\n```\n# Clone vllm to get the testing script (if not already present)\ngit clone https://github.com/vllm-project/vllm.git\ncd vllm\n\n# Run the proxy, pointing it to both workers\npython tests/disagg_prefill/toy_proxy_server.py \\\n  --prefill http://10.0.0.1:8200 \\\n  --decode http://10.0.0.2:8201 \\\n  --port 8000\n```\n\n*Note: For production, replace `toy_proxy_server.py` with a robust P/D routing solution like NVIDIA Dynamo or `llm-d`.*\n\nYou can now send standard OpenAI-compatible requests to the proxy on port `8000`. The proxy will seamlessly split the workload, leveraging the massive VRAM and compute of your RTX PRO 6000s.\n\n```\ncurl http://localhost:8000/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"meta-llama/Llama-3.1-70B-Instruct\",\n    \"messages\": [\n      {\"role\": \"system\", \"content\": \"You are a helpful AI assistant.\"},\n      {\"role\": \"user\", \"content\": \"Explain the architectural advantages of the Blackwell SM120 GPU.\"}\n    ],\n    \"max_tokens\": 1024\n  }'\n```\n\n", "url": "https://wpnews.pro/news/rtx6000-pd-md", "canonical_source": "https://gist.github.com/tsg-/7148a5da0b21333b62a98f41b486ff6d", "published_at": "2026-11-10 21:56:49+00:00", "updated_at": "2026-11-10 22:23:05.861435+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "mlops"], "entities": ["NVIDIA", "RTX PRO 6000", "vLLM", "NixlConnector", "NVIDIA Dynamo", "llm-d", "Mellanox", "Meta Llama 3.1"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/rtx6000-pd-md", "markdown": "https://wpnews.pro/news/rtx6000-pd-md.md", "text": "https://wpnews.pro/news/rtx6000-pd-md.txt", "jsonld": "https://wpnews.pro/news/rtx6000-pd-md.jsonld"}}