{"slug": "nvidia-dynamo-the-orchestration-layer-that-turns-vllm-into-a-cluster", "title": "NVIDIA Dynamo: The Orchestration Layer That Turns vLLM Into a Cluster", "summary": "NVIDIA Dynamo, an Apache-2.0 datacenter-scale inference orchestration layer from NVIDIA, sits above vLLM, SGLang, and TensorRT-LLM to coordinate multi-GPU clusters, with published benchmarks showing up to 750x higher throughput on GB300 NVL72. The project, built in Rust and Python with over 7,600 GitHub stars and 160+ contributors, enables disaggregated prefill/decode, KV-aware routing, multi-tier KV caching, and SLA-driven autoscaling, while the README notes that a single GPU setup may not need it.", "body_md": "# NVIDIA Dynamo: The Orchestration Layer That Turns vLLM Into a Cluster\n\nNVIDIA Dynamo is an Apache-2.0 datacenter-scale inference orchestration layer that sits above vLLM, SGLang, and TensorRT-LLM — disaggregated prefill/decode, KV-aware routing, multi-tier KV caching, and SLA-driven autoscaling, with published benchmarks up to 750x higher throughput on GB300 NVL72.\n\n- ⭐ 7658\n- Rust\n- Python\n- Apache-2.0\n- Updated 2026-08-03\n\n[SGLang: Structured Generation for LLMs](https://dibi8.com/resources/llm-frameworks/sglang-structured-generation-llm/) •\n[Self-Hosted LLM 2026: Ollama vs vLLM vs LocalAI](https://dibi8.com/resources/llm-frameworks/self-hosted-llm-2026-ollama-vllm-localai/)\n\n*Project banner — from github.com/ai-dynamo/dynamo*\n\n## What Is NVIDIA Dynamo? [#](#what-is-nvidia-dynamo)\n\n**Dynamo** is NVIDIA’s open-source, datacenter-scale inference stack — but the framing that matters most is what it *isn’t*: it doesn’t replace SGLang, TensorRT-LLM, or vLLM. It’s **the orchestration layer above them**, turning a cluster of GPUs running any of those engines into a coordinated multi-node inference system, with disaggregated serving, intelligent routing, multi-tier KV caching, and automatic scaling working together.\n\n🔗 **GitHub**: [https://github.com/ai-dynamo/dynamo](https://github.com/ai-dynamo/dynamo)\n📖 **Docs**: [docs.nvidia.com/dynamo](https://docs.nvidia.com/dynamo/)\n\nApache-2.0 licensed, built in **Rust for performance, Python for extensibility**, at **7,600+ GitHub stars** with **160+ community contributors**, and a commit from **August 1, 2026** — the day before this article.\n\n## When to Use It (and When Not To) [#](#when-to-use-it-and-when-not-to)\n\nThe README is upfront about the boundary:\n\n- You’re serving LLMs across\n**multiple GPUs or nodes** and need to coordinate them - You want\n**KV-aware routing** to avoid redundant prefill computation - You need to\n**independently scale prefill and decode**(disaggregated serving) - You want\n**automatic scaling** that meets latency SLAs at minimum total cost of ownership - You need\n**fast cold-starts** when spinning up new replicas\n\n**“If you’re running a single model on a single GPU, your inference engine alone is probably sufficient.”** — that’s not a hedge, it’s the project telling you when not to add the complexity.\n\n## Backend Support [#](#backend-support)\n\n| Feature | SGLang | TensorRT-LLM | vLLM |\n|---|---|---|---|\n| Disaggregated Serving | ✅ | ✅ | ✅ |\n| KV-Aware Routing | ✅ | ✅ | ✅ |\n| SLA-Based Planner | ✅ | ✅ | ✅ |\n| KVBM (multi-tier KV cache) | 🚧 | ✅ | ✅ |\n| Multimodal | ✅ | ✅ | ✅ |\n| Tool Calling | ✅ | ✅ | ✅ |\n\n## Published Results (Each Sourced, Not Just Asserted) [#](#published-results-each-sourced-not-just-asserted)\n\n| Result | Context |\n|---|---|\n7× higher throughput per GPU | DeepSeek R1 on GB200 NVL72 with Dynamo vs. B200 without (per\n|\n\n**7×** faster model startup**2×** faster time to first token[Baseten benchmark](https://www.baseten.co/blog/how-baseten-achieved-2x-faster-inference-with-nvidia-dynamo/))**80%** fewer SLA breaches**750×** higher throughputEach of these is attributed to a specific benchmark or source rather than presented as an unsourced headline number — worth checking the linked methodology before treating any single figure as your expected result.\n\n## Core Capabilities [#](#core-capabilities)\n\n| Capability | What it does | Why it matters |\n|---|---|---|\nDisaggregated Prefill/Decode | Separates prefill and decode into independently scalable GPU pools | Each phase runs on hardware tuned for its workload |\nKV-Aware Routing | Routes requests based on worker load and KV cache overlap | Eliminates redundant prefill — the 2× TTFT improvement above |\nKV Block Manager (KVBM) | Offloads KV cache across GPU → CPU → SSD → remote storage | Extends effective context length beyond raw GPU memory |\nModelExpress | Streams model weights GPU-to-GPU via NIXL/NVLink | 7× faster cold-start for new replicas |\nPlanner | SLA-driven autoscaler that profiles workloads and right-sizes pools | Meets latency targets at minimum TCO |\nGrove | Kubernetes operator for topology-aware gang scheduling (NVL72) | Places workloads optimally across racks, hosts, NUMA nodes |\nAIConfigurator | Simulates 10,000+ deployment configs in seconds | Finds an optimal serving config without burning GPU-hours on trial and error |\nFault Tolerance | Canary health checks + in-flight request migration | Workers can fail without failing user requests |\n\n### New in 1.0 [#](#new-in-10)\n\n**Zero-config deploy (DGDR, beta)**— specify model, hardware, and SLA in one YAML; AIConfigurator profiles the workload and Planner optimizes topology automatically**Agentic inference**— per-request hints for priority, expected output length, speculative prefill, plus LangChain and NeMo Agent Toolkit integrations**Multimodal E/P/D**— disaggregated encode/prefill/decode with embedding cache, 30% faster TTFT on image workloads** Video generation**— native FastVideo + SGLang Diffusion support, real-time 1080p on a single B200** K8s Inference Gateway plugin**— KV-aware routing inside the standard Kubernetes gateway\n\n## Quick Start [#](#quick-start)\n\n### Container (fastest) [#](#container-fastest)\n\n```\ndocker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.3.0\n# Inside the container — start frontend and worker\npython3 -m dynamo.frontend --http-port 8000 --discovery-backend file > /dev/null 2>&1 &\npython3 -m dynamo.sglang --model-path Qwen/Qwen3-0.6B --discovery-backend file &\ncurl -s localhost:8000/v1/chat/completions -H \"Content-Type: application/json\" -d '{\n  \"model\": \"Qwen/Qwen3-0.6B\",\n  \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}],\n  \"max_tokens\": 100\n}' | jq\n```\n\n`tensorrtllm-runtime:1.3.0`\n\nand `vllm-runtime:1.3.0`\n\ncontainers are available the same way.\n\n### PyPI install [#](#pypi-install)\n\n```\nuv pip install --prerelease=allow \"ai-dynamo[sglang]\"   # or [vllm]\n```\n\n### Kubernetes (recommended for production) [#](#kubernetes-recommended-for-production)\n\n```\n# Zero-config deploy: specify model + SLA, Dynamo handles the rest\napiVersion: nvidia.com/v1beta1\nkind: DynamoGraphDeploymentRequest\nmetadata:\n  name: my-model\nspec:\n  model: Qwen/Qwen3-0.6B\n  backend: vllm\n  sla:\n    ttft: 200.0   # ms\n    itl: 20.0     # ms\n  autoApply: true\n```\n\nPre-built recipes exist for common models (Llama-3-70B on vLLM, DeepSeek-R1 on SGLang disaggregated, Qwen3-32B-FP8 on TensorRT-LLM), plus cloud-specific guides for AWS EKS, Google GKE, Azure AKS, and Amazon ECS.\n\n## Two Request-Routing Topologies [#](#two-request-routing-topologies)\n\n| Topology | What it is | When to use |\n|---|---|---|\nDynamo-native Frontend routing | Dynamo’s own Frontend + Router make worker-selection decisions, no external gateway needed | Local development, single-cluster deployments |\nGateway API routing (GAIE) | A Kubernetes Gateway API Inference Extension calls Dynamo’s Endpoint Picker Plugin before forwarding | Platforms standardizing on Gateway API, where policy/auth/rate-limiting should sit at the cluster edge |\n\nBoth expose an OpenAI-compatible API and support the same backends, disaggregated serving, and KV-aware routing — the choice is about where routing decisions live in your infrastructure, not a feature trade-off.\n\n## Use Cases [#](#use-cases)\n\n### 1. Scaling Past a Single Node [#](#1-scaling-past-a-single-node)\n\nOnce one GPU (or one node) can’t serve your traffic, Dynamo coordinates multiple nodes as one inference system instead of you hand-rolling load balancing across independent engine instances.\n\n### 2. Cutting Redundant Prefill Compute [#](#2-cutting-redundant-prefill-compute)\n\nKV-aware routing avoids recomputing prefill for requests that hit workers already holding relevant KV cache — directly behind the published 2× TTFT improvement.\n\n### 3. Meeting Latency SLAs Without Overprovisioning [#](#3-meeting-latency-slas-without-overprovisioning)\n\nThe Planner’s SLA-driven autoscaling targets a specific TTFT/ITL budget rather than scaling on raw CPU/GPU utilization, aimed at the 80% fewer SLA breaches at lower TCO result.\n\n### 4. Serving Reasoning, Multimodal, or Video Workloads at Scale [#](#4-serving-reasoning-multimodal-or-video-workloads-at-scale)\n\nNative support for disaggregated multimodal encode/prefill/decode and video generation (FastVideo, SGLang Diffusion) extends beyond text-only LLM serving.\n\n## Related Repositories [#](#related-repositories)\n\n| Repository | Purpose |\n|---|---|\n|\n\n[Grove](https://github.com/ai-dynamo/grove)[AIConfigurator](https://github.com/ai-dynamo/aiconfigurator)## Related Articles [#](#related-articles)\n\n[SGLang: Structured Generation for LLMs](https://dibi8.com/resources/llm-frameworks/sglang-structured-generation-llm/)— one of the three inference engines Dynamo orchestrates rather than replaces[Self-Hosted LLM 2026: Ollama vs vLLM vs LocalAI](https://dibi8.com/resources/llm-frameworks/self-hosted-llm-2026-ollama-vllm-localai/)— for single-node self-hosting, the scale below where Dynamo’s orchestration layer becomes relevant\n\n## Conclusion [#](#conclusion)\n\n**NVIDIA Dynamo** targets a specific, well-defined gap: individual inference engines are excellent at serving one model efficiently on one GPU or node, but coordinating dozens of GPUs — disaggregating prefill from decode, routing around redundant KV computation, autoscaling to an SLA rather than a CPU graph — is a separate, harder problem most engines don’t solve themselves. Apache-2.0 licensed, backed by NVIDIA, and unusually disciplined about sourcing its own performance claims, it’s a serious option once you’ve outgrown “one engine, one GPU” — and the README itself tells you plainly when you haven’t.\n\n**Best for**: Teams running LLM inference across multiple GPUs or nodes who need disaggregated serving, KV-aware routing, or SLA-driven autoscaling on top of vLLM, SGLang, or TensorRT-LLM — not single-GPU deployments.\n\n**GitHub**: [https://github.com/ai-dynamo/dynamo](https://github.com/ai-dynamo/dynamo)\n\n*Last updated: 2026-08-03*", "url": "https://wpnews.pro/news/nvidia-dynamo-the-orchestration-layer-that-turns-vllm-into-a-cluster", "canonical_source": "https://dibi8.com/resources/llm-frameworks/nvidia-dynamo-distributed-inference-2026/", "published_at": "2026-08-02 17:55:00+00:00", "updated_at": "2026-08-02 19:36:01.049288+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-infrastructure", "ai-tools", "ai-research"], "entities": ["NVIDIA", "Dynamo", "vLLM", "SGLang", "TensorRT-LLM", "Baseten", "GB300 NVL72", "GB200 NVL72"], "alternates": {"html": "https://wpnews.pro/news/nvidia-dynamo-the-orchestration-layer-that-turns-vllm-into-a-cluster", "markdown": "https://wpnews.pro/news/nvidia-dynamo-the-orchestration-layer-that-turns-vllm-into-a-cluster.md", "text": "https://wpnews.pro/news/nvidia-dynamo-the-orchestration-layer-that-turns-vllm-into-a-cluster.txt", "jsonld": "https://wpnews.pro/news/nvidia-dynamo-the-orchestration-layer-that-turns-vllm-into-a-cluster.jsonld"}}