NVIDIA Dynamo: The Orchestration Layer That Turns vLLM Into a Cluster

NVIDIA Dynamo, an Apache-2.0 datacenter-scale inference orchestration layer from NVIDIA, sits above vLLM, SGLang, and TensorRT-LLM to coordinate multi-GPU clusters, with published benchmarks showing up to 750x higher throughput on GB300 NVL72. The project, built in Rust and Python with over 7,600 GitHub stars and 160+ contributors, enables disaggregated prefill/decode, KV-aware routing, multi-tier KV caching, and SLA-driven autoscaling, while the README notes that a single GPU setup may not need it.

NVIDIA Dynamo: The Orchestration Layer That Turns vLLM Into a Cluster NVIDIA Dynamo is an Apache-2.0 datacenter-scale inference orchestration layer that sits above vLLM, SGLang, and TensorRT-LLM — disaggregated prefill/decode, KV-aware routing, multi-tier KV caching, and SLA-driven autoscaling, with published benchmarks up to 750x higher throughput on GB300 NVL72. - ⭐ 7658 - Rust - Python - Apache-2.0 - Updated 2026-08-03 SGLang: Structured Generation for LLMs https://dibi8.com/resources/llm-frameworks/sglang-structured-generation-llm/ • Self-Hosted LLM 2026: Ollama vs vLLM vs LocalAI https://dibi8.com/resources/llm-frameworks/self-hosted-llm-2026-ollama-vllm-localai/ Project banner — from github.com/ai-dynamo/dynamo What Is NVIDIA Dynamo? what-is-nvidia-dynamo Dynamo is NVIDIA’s open-source, datacenter-scale inference stack — but the framing that matters most is what it isn’t : it doesn’t replace SGLang, TensorRT-LLM, or vLLM. It’s the orchestration layer above them , turning a cluster of GPUs running any of those engines into a coordinated multi-node inference system, with disaggregated serving, intelligent routing, multi-tier KV caching, and automatic scaling working together. 🔗 GitHub : https://github.com/ai-dynamo/dynamo https://github.com/ai-dynamo/dynamo 📖 Docs : docs.nvidia.com/dynamo https://docs.nvidia.com/dynamo/ Apache-2.0 licensed, built in Rust for performance, Python for extensibility , at 7,600+ GitHub stars with 160+ community contributors , and a commit from August 1, 2026 — the day before this article. When to Use It and When Not To when-to-use-it-and-when-not-to The README is upfront about the boundary: - You’re serving LLMs across multiple GPUs or nodes and need to coordinate them - You want KV-aware routing to avoid redundant prefill computation - You need to independently scale prefill and decode disaggregated serving - You want automatic scaling that meets latency SLAs at minimum total cost of ownership - You need fast cold-starts when spinning up new replicas “If you’re running a single model on a single GPU, your inference engine alone is probably sufficient.” — that’s not a hedge, it’s the project telling you when not to add the complexity. Backend Support backend-support | Feature | SGLang | TensorRT-LLM | vLLM | |---|---|---|---| | Disaggregated Serving | ✅ | ✅ | ✅ | | KV-Aware Routing | ✅ | ✅ | ✅ | | SLA-Based Planner | ✅ | ✅ | ✅ | | KVBM multi-tier KV cache | 🚧 | ✅ | ✅ | | Multimodal | ✅ | ✅ | ✅ | | Tool Calling | ✅ | ✅ | ✅ | Published Results Each Sourced, Not Just Asserted published-results-each-sourced-not-just-asserted | Result | Context | |---|---| 7× higher throughput per GPU | DeepSeek R1 on GB200 NVL72 with Dynamo vs. B200 without per | 7× faster model startup 2× faster time to first token Baseten benchmark https://www.baseten.co/blog/how-baseten-achieved-2x-faster-inference-with-nvidia-dynamo/ 80% fewer SLA breaches 750× higher throughputEach of these is attributed to a specific benchmark or source rather than presented as an unsourced headline number — worth checking the linked methodology before treating any single figure as your expected result. Core Capabilities core-capabilities | Capability | What it does | Why it matters | |---|---|---| Disaggregated Prefill/Decode | Separates prefill and decode into independently scalable GPU pools | Each phase runs on hardware tuned for its workload | KV-Aware Routing | Routes requests based on worker load and KV cache overlap | Eliminates redundant prefill — the 2× TTFT improvement above | KV Block Manager KVBM | Offloads KV cache across GPU → CPU → SSD → remote storage | Extends effective context length beyond raw GPU memory | ModelExpress | Streams model weights GPU-to-GPU via NIXL/NVLink | 7× faster cold-start for new replicas | Planner | SLA-driven autoscaler that profiles workloads and right-sizes pools | Meets latency targets at minimum TCO | Grove | Kubernetes operator for topology-aware gang scheduling NVL72 | Places workloads optimally across racks, hosts, NUMA nodes | AIConfigurator | Simulates 10,000+ deployment configs in seconds | Finds an optimal serving config without burning GPU-hours on trial and error | Fault Tolerance | Canary health checks + in-flight request migration | Workers can fail without failing user requests | New in 1.0 new-in-10 Zero-config deploy DGDR, beta — specify model, hardware, and SLA in one YAML; AIConfigurator profiles the workload and Planner optimizes topology automatically Agentic inference — per-request hints for priority, expected output length, speculative prefill, plus LangChain and NeMo Agent Toolkit integrations Multimodal E/P/D — disaggregated encode/prefill/decode with embedding cache, 30% faster TTFT on image workloads Video generation — native FastVideo + SGLang Diffusion support, real-time 1080p on a single B200 K8s Inference Gateway plugin — KV-aware routing inside the standard Kubernetes gateway Quick Start quick-start Container fastest container-fastest docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.3.0 Inside the container — start frontend and worker python3 -m dynamo.frontend --http-port 8000 --discovery-backend file /dev/null 2 &1 & python3 -m dynamo.sglang --model-path Qwen/Qwen3-0.6B --discovery-backend file & curl -s localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen3-0.6B", "messages": {"role": "user", "content": "Hello "} , "max tokens": 100 }' | jq tensorrtllm-runtime:1.3.0 and vllm-runtime:1.3.0 containers are available the same way. PyPI install pypi-install uv pip install --prerelease=allow "ai-dynamo sglang " or vllm Kubernetes recommended for production kubernetes-recommended-for-production Zero-config deploy: specify model + SLA, Dynamo handles the rest apiVersion: nvidia.com/v1beta1 kind: DynamoGraphDeploymentRequest metadata: name: my-model spec: model: Qwen/Qwen3-0.6B backend: vllm sla: ttft: 200.0 ms itl: 20.0 ms autoApply: true Pre-built recipes exist for common models Llama-3-70B on vLLM, DeepSeek-R1 on SGLang disaggregated, Qwen3-32B-FP8 on TensorRT-LLM , plus cloud-specific guides for AWS EKS, Google GKE, Azure AKS, and Amazon ECS. Two Request-Routing Topologies two-request-routing-topologies | Topology | What it is | When to use | |---|---|---| Dynamo-native Frontend routing | Dynamo’s own Frontend + Router make worker-selection decisions, no external gateway needed | Local development, single-cluster deployments | Gateway API routing GAIE | A Kubernetes Gateway API Inference Extension calls Dynamo’s Endpoint Picker Plugin before forwarding | Platforms standardizing on Gateway API, where policy/auth/rate-limiting should sit at the cluster edge | Both expose an OpenAI-compatible API and support the same backends, disaggregated serving, and KV-aware routing — the choice is about where routing decisions live in your infrastructure, not a feature trade-off. Use Cases use-cases 1. Scaling Past a Single Node 1-scaling-past-a-single-node Once one GPU or one node can’t serve your traffic, Dynamo coordinates multiple nodes as one inference system instead of you hand-rolling load balancing across independent engine instances. 2. Cutting Redundant Prefill Compute 2-cutting-redundant-prefill-compute KV-aware routing avoids recomputing prefill for requests that hit workers already holding relevant KV cache — directly behind the published 2× TTFT improvement. 3. Meeting Latency SLAs Without Overprovisioning 3-meeting-latency-slas-without-overprovisioning The Planner’s SLA-driven autoscaling targets a specific TTFT/ITL budget rather than scaling on raw CPU/GPU utilization, aimed at the 80% fewer SLA breaches at lower TCO result. 4. Serving Reasoning, Multimodal, or Video Workloads at Scale 4-serving-reasoning-multimodal-or-video-workloads-at-scale Native support for disaggregated multimodal encode/prefill/decode and video generation FastVideo, SGLang Diffusion extends beyond text-only LLM serving. Related Repositories related-repositories | Repository | Purpose | |---|---| | Grove https://github.com/ai-dynamo/grove AIConfigurator https://github.com/ai-dynamo/aiconfigurator Related Articles related-articles SGLang: Structured Generation for LLMs https://dibi8.com/resources/llm-frameworks/sglang-structured-generation-llm/ — one of the three inference engines Dynamo orchestrates rather than replaces Self-Hosted LLM 2026: Ollama vs vLLM vs LocalAI https://dibi8.com/resources/llm-frameworks/self-hosted-llm-2026-ollama-vllm-localai/ — for single-node self-hosting, the scale below where Dynamo’s orchestration layer becomes relevant Conclusion conclusion NVIDIA Dynamo targets a specific, well-defined gap: individual inference engines are excellent at serving one model efficiently on one GPU or node, but coordinating dozens of GPUs — disaggregating prefill from decode, routing around redundant KV computation, autoscaling to an SLA rather than a CPU graph — is a separate, harder problem most engines don’t solve themselves. Apache-2.0 licensed, backed by NVIDIA, and unusually disciplined about sourcing its own performance claims, it’s a serious option once you’ve outgrown “one engine, one GPU” — and the README itself tells you plainly when you haven’t. Best for : Teams running LLM inference across multiple GPUs or nodes who need disaggregated serving, KV-aware routing, or SLA-driven autoscaling on top of vLLM, SGLang, or TensorRT-LLM — not single-GPU deployments. GitHub : https://github.com/ai-dynamo/dynamo https://github.com/ai-dynamo/dynamo Last updated: 2026-08-03