Deploying Inference Using NVIDIA Dynamo and vLLM

NVIDIA Dynamo, an open-source inference framework, has been deployed with the vLLM backend to serve chat completion requests in both aggregated and disaggregated configurations. The deployment involves setting up infrastructure services like etcd and NATS, pulling a CUDA-matched vLLM container, and running the Dynamo repository's release 0.9.0. This setup enables high-throughput, low-latency inference for large-scale generative AI models across multi-GPU environments.

NVIDIA Dynamo https://docs.nvidia.com/dynamo/getting-started/quickstart is an open-source, high-throughput, low-latency inference framework for deploying large-scale generative AI and reasoning models across multi-node, multi-GPU environments. It boosts LLM inference efficiency and reduces time-to-first-token TTFT through intelligent resource management, dynamic GPU allocation, and distributed orchestration, and it supports disaggregated serving — separating prompt processing prefill and response generation decode across different GPUs so each phase can be optimized independently. Dynamo integrates with backends including vLLM, SGLang, and NVIDIA TensorRT-LLM. This guide deploys NVIDIA Dynamo with the vLLM backend https://docs.nvidia.com/dynamo/components/backends/v-llm , covering infrastructure setup, container deployment, and two serving patterns: aggregated serving for single-GPU configurations and disaggregated serving for multi-GPU setups with independent prefill and decode workers. By the end, you'll have a running Dynamo + vLLM deployment serving chat completion requests in both aggregated and disaggregated configurations. NVIDIA Dynamo provides deployment scripts, container utilities, and orchestration modules required to run inference workloads. 1. Clone the repository: bash $ git clone https://github.com/ai-dynamo/dynamo.git 2. Navigate to the repository directory: bash $ cd dynamo 3. Switch to the latest stable release: bash $ git checkout release/0.9.0 Visit the Dynamo releases page https://github.com/ai-dynamo/dynamo/releases/ to find the latest stable release version. Dynamo's distributed architecture relies on etcd for worker registry and service discovery, and NATS for KV cache event propagation between prefill and decode workers. The Docker Compose configuration launches both with exposed ports for client connections etcd: 2379-2380, NATS: 4222, 6222, 8222 . These services must run continuously for Dynamo to coordinate worker resources and route inference requests. 1. Start the infrastructure services: bash $ docker compose -f deploy/docker-compose.yml up -d 2. Verify the services are running: bash $ docker compose -f deploy/docker-compose.yml ps The output displays the running etcd and NATS containers: NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS deploy-etcd-server-1 bitnamilegacy/etcd:3.6.1 "/opt/bitnami/script…" etcd-server 6 seconds ago Up 6 seconds 0.0.0.0:2379-2380- 2379-2380/tcp, :: :2379-2380- 2379-2380/tcp deploy-nats-server-1 nats:2.11.4 "/nats-server -c /et…" nats-server 6 seconds ago Up 6 seconds 0.0.0.0:4222- 4222/tcp, :: :4222- 4222/tcp, 0.0.0.0:6222- 6222/tcp, :: :6222- 6222/tcp, 0.0.0.0:8222- 8222/tcp, :: :8222- 8222/tcp The vLLM container requires a CUDA version match between the host driver and container runtime to prevent GPU kernel incompatibilities. 1. Check the installed CUDA version: bash $ nvidia-smi The output displays the CUDA version in the top-right corner of the table: +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 580.95.05 Driver Version: 580.95.05 CUDA Version: 13.0 | +-----------------------------------------+------------------------+----------------------+ .... 2. Pull the vLLM container image from NGC, matching the image tag to your system's CUDA version: For CUDA 13.x: bash $ docker pull nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13 For CUDA 12.x: bash $ docker pull nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0 Visit the NVIDIA NGC Catalog to view all available image tags and CUDA versions. 3. Optional Build the container from source instead of pulling the pre-built image: $ ./container/build.sh --framework VLLM This creates an image named dynamo:latest-vllm . If you use this locally built image, replace nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13 with dynamo:latest-vllm in all subsequent commands. The container runs as UID 1000 and requires write access to the Hugging Face cache directory for model downloads. Incorrect permissions prevent the container from accessing cached model weights, causing worker initialization failures. 1. Create the cache directory if it does not exist: bash $ mkdir -p container/.cache/huggingface 2. Set ownership to the container user UID 1000 : bash $ sudo chown -R 1000:1000 container/.cache/huggingface 3. Set appropriate permissions: bash $ sudo chmod -R 775 container/.cache/huggingface Aggregated serving https://docs.nvidia.com/dynamo/components/backends/v-llm aggregated-serving combines prefill and decode phases on a single worker, eliminating inter-GPU KV cache transfers and reducing request latency. This suits single-GPU environments or workloads prioritizing response time over throughput. 1. Export your Hugging Face token to avoid rate limitations when downloading large models replace YOUR HF TOKEN with your actual token : bash $ export HF TOKEN=YOUR HF TOKEN 2. Run the vLLM container with GPU access and workspace mounting, using the image tag that matches your CUDA version: $ ./container/run.sh -it --framework VLLM --mount-workspace --image nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13 -e HF TOKEN=$HF TOKEN 3. Inside the container, create a custom launch script for aggregated serving with the NVIDIA Nemotron model: bash $ cat << 'EOF' ~/nemotron agg.sh /bin/bash set -e trap 'echo Cleaning up...; kill 0' EXIT Set deterministic hash for KV event IDs export PYTHONHASHSEED=0 Model configuration - use command line argument or default MODEL="${1:-nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1}" echo "Starting Dynamo Frontend..." python -m dynamo.frontend & echo "Starting vLLM Worker with model: $MODEL" DYN SYSTEM PORT=${DYN SYSTEM PORT:-8081} \ python -m dynamo.vllm \ --model "$MODEL" \ --trust-remote-code \ --enforce-eager \ --connector none EOF 4. Make the script executable: bash $ chmod +x ~/nemotron agg.sh 5. Run the aggregated serving script: $ ~/nemotron agg.sh This starts a frontend service on port 8000 and a vLLM worker that loads the specified model defaults to NVIDIA Nemotron Nano 4B . To deploy the larger NVIDIA Nemotron Super 49B model instead, pass the model name as an argument: $ ~/nemotron agg.sh "nvidia/Llama-3 3-Nemotron-Super-49B-v1 5" The 49B model requires high-memory GPUs such as B200 or GB200 class devices. Ensure sufficient VRAM and consider tensor parallelism for production deployments. 6. Open a new terminal session on your server outside the container and test with a chat completion request: bash $ curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1", "messages": {"role": "user", "content": "Hello Tell me about AI."} , "max tokens": 100 }' The output displays the model's chat response in JSON format. Disaggregated serving https://docs.nvidia.com/dynamo/user-guides/disaggregated-serving assigns prefill and decode phases to separate GPU workers, enabling independent scaling and optimization of each phase. Prefill workers process incoming prompts and transfer KV cache data to decode workers via NIXL. This maximizes throughput by letting multiple decode workers share prefill resources, improving GPU utilization across the cluster. 1. Exit the container if you are still inside from the previous section — press Ctrl+C to terminate the running process, then Ctrl+D to exit. 2. Export your Hugging Face token replace YOUR HF TOKEN with your actual token : bash $ export HF TOKEN=YOUR HF TOKEN 3. Run the container with the image tag that matches your CUDA version: $ ./container/run.sh -it --framework VLLM --mount-workspace --image nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13 -e HF TOKEN=$HF TOKEN 4. Inside the container, create a custom launch script for disaggregated serving with the NVIDIA Nemotron model: bash $ cat << 'EOF' ~/nemotron disagg.sh /bin/bash Kill any existing processes pkill -f "dynamo.frontend" pkill -f "dynamo.vllm" sleep 2 Set deterministic hash for KV event IDs export PYTHONHASHSEED=0 Model configuration - use command line argument or default MODEL="${1:-nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1}" echo "Starting Dynamo Frontend..." python -m dynamo.frontend & echo "Starting Decode Workers with model: $MODEL" Decode Worker 1 - Uses GPU 0 CUDA VISIBLE DEVICES=0 python3 -m dynamo.vllm \ --model "$MODEL" \ --trust-remote-code \ --is-decode-worker \ --max-model-len 2048 & Decode Worker 2 - Uses GPU 1 VLLM NIXL SIDE CHANNEL PORT=20097 \ CUDA VISIBLE DEVICES=1 python3 -m dynamo.vllm \ --model "$MODEL" \ --trust-remote-code \ --is-decode-worker \ --max-model-len 2048 & echo "Starting Prefill Workers with model: $MODEL" Prefill Worker 1 - Uses GPU 2 CUDA VISIBLE DEVICES=2 python3 -m dynamo.vllm \ --model "$MODEL" \ --trust-remote-code \ --is-prefill-worker \ --max-model-len 2048 \ --kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp:// :20082","enable kv cache events":true}' & Prefill Worker 2 - Uses GPU 3 VLLM NIXL SIDE CHANNEL PORT=20099 \ CUDA VISIBLE DEVICES=3 python3 -m dynamo.vllm \ --model "$MODEL" \ --trust-remote-code \ --is-prefill-worker \ --max-model-len 2048 \ --kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp:// :20083","enable kv cache events":true}' & echo "All services starting... waiting for initialization..." sleep 30 echo "Deployment ready " Keep script running wait EOF 5. Make the script executable: bash $ chmod +x ~/nemotron disagg.sh 6. Run the disaggregated serving script: $ ~/nemotron disagg.sh This starts the frontend service on port 8000 , two decode workers on GPUs 0 and 1 , and two prefill workers on GPUs 2 and 3 with the specified model defaults to NVIDIA Nemotron Nano 4B . To deploy the larger NVIDIA Nemotron Super 49B model instead, pass the model name as an argument: $ ~/nemotron disagg.sh "nvidia/Llama-3 3-Nemotron-Super-49B-v1 5" The 49B model requires high-memory GPUs such as B200 or GB200 class devices. Ensure sufficient VRAM and consider tensor parallelism for production deployments. 7. Open a new terminal session on your server outside the container and test with multiple sequential requests to observe worker distribution: bash $ for i in {1..5}; do echo "Request $i:" curl -s http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d "{ \"model\": \"nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1\", \"messages\": {\"role\": \"user\", \"content\": \"Test request $i\"} , \"max tokens\": 10 }" | jq '.id' sleep 1 done Each request returns a unique ID, and the logs inside the container show which workers process each request. 8. Test with concurrent requests to verify load distribution: bash $ for i in {1..10}; do curl -s http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d "{ \"model\": \"nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1\", \"messages\": {\"role\": \"user\", \"content\": \"Concurrent test $i\"} , \"max tokens\": 20 }" & done wait echo "All requests completed" Dynamo's router distributes the requests across available prefill and decode workers. For the full guide with additional tips, visit the original article on Vultr Docs .