NVIDIA Dynamo is an Apache-2.0 datacenter-scale inference orchestration layer that sits above vLLM, SGLang, and TensorRT-LLM — disaggregated prefill/decode, KV-aware routing, multi-tier KV caching, and SLA-driven autoscaling, with published benchmarks up to 750x higher throughput on GB300 NVL72.
- ⭐ 7658
- Rust
- Python
- Apache-2.0
- Updated 2026-08-03
SGLang: Structured Generation for LLMs • Self-Hosted LLM 2026: Ollama vs vLLM vs LocalAI
Project banner — from github.com/ai-dynamo/dynamo
What Is NVIDIA Dynamo? # #
Dynamo is NVIDIA’s open-source, datacenter-scale inference stack — but the framing that matters most is what it isn’t: it doesn’t replace SGLang, TensorRT-LLM, or vLLM. It’s the orchestration layer above them, turning a cluster of GPUs running any of those engines into a coordinated multi-node inference system, with disaggregated serving, intelligent routing, multi-tier KV caching, and automatic scaling working together.
🔗 GitHub: https://github.com/ai-dynamo/dynamo 📖 Docs: docs.nvidia.com/dynamo
Apache-2.0 licensed, built in Rust for performance, Python for extensibility, at 7,600+ GitHub stars with 160+ community contributors, and a commit from August 1, 2026 — the day before this article.
When to Use It (and When Not To) # #
The README is upfront about the boundary:
- You’re serving LLMs across multiple GPUs or nodes and need to coordinate them - You want KV-aware routing to avoid redundant prefill computation - You need to independently scale prefill and decode(disaggregated serving) - You want automatic scaling that meets latency SLAs at minimum total cost of ownership - You need fast cold-starts when spinning up new replicas
“If you’re running a single model on a single GPU, your inference engine alone is probably sufficient.” — that’s not a hedge, it’s the project telling you when not to add the complexity.
Backend Support # #
| Feature | SGLang | TensorRT-LLM | vLLM |
|---|---|---|---|
| Disaggregated Serving | ✅ | ✅ | ✅ |
| KV-Aware Routing | ✅ | ✅ | ✅ |
| SLA-Based Planner | ✅ | ✅ | ✅ |
| KVBM (multi-tier KV cache) | 🚧 | ✅ | ✅ |
| Multimodal | ✅ | ✅ | ✅ |
| Tool Calling | ✅ | ✅ | ✅ |
Published Results (Each Sourced, Not Just Asserted) # #
| Result | Context |
|---|---|
| 7× higher throughput per GPU | DeepSeek R1 on GB200 NVL72 with Dynamo vs. B200 without (per |
7× faster model startup2× faster time to first tokenBaseten benchmark)80% fewer SLA breaches750× higher throughputEach of these is attributed to a specific benchmark or source rather than presented as an unsourced headline number — worth checking the linked methodology before treating any single figure as your expected result.
Core Capabilities # #
| Capability | What it does | Why it matters |
|---|---|---|
| Disaggregated Prefill/Decode | Separates prefill and decode into independently scalable GPU pools | Each phase runs on hardware tuned for its workload |
| KV-Aware Routing | Routes requests based on worker load and KV cache overlap | Eliminates redundant prefill — the 2× TTFT improvement above |
| KV Block Manager (KVBM) | Offloads KV cache across GPU → CPU → SSD → remote storage | Extends effective context length beyond raw GPU memory |
| ModelExpress | Streams model weights GPU-to-GPU via NIXL/NVLink | 7× faster cold-start for new replicas |
| Planner | SLA-driven autoscaler that profiles workloads and right-sizes pools | Meets latency targets at minimum TCO |
| Grove | Kubernetes operator for topology-aware gang scheduling (NVL72) | Places workloads optimally across racks, hosts, NUMA nodes |
| AIConfigurator | Simulates 10,000+ deployment configs in seconds | Finds an optimal serving config without burning GPU-hours on trial and error |
| Fault Tolerance | Canary health checks + in-flight request migration | Workers can fail without failing user requests |
New in 1.0 #
Zero-config deploy (DGDR, beta)— specify model, hardware, and SLA in one YAML; AIConfigurator profiles the workload and Planner optimizes topology automaticallyAgentic inference— per-request hints for priority, expected output length, speculative prefill, plus LangChain and NeMo Agent Toolkit integrationsMultimodal E/P/D— disaggregated encode/prefill/decode with embedding cache, 30% faster TTFT on image workloads** Video generation**— native FastVideo + SGLang Diffusion support, real-time 1080p on a single B200** K8s Inference Gateway plugin**— KV-aware routing inside the standard Kubernetes gateway
Quick Start # #
Container (fastest) #
docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.3.0
python3 -m dynamo.frontend --http-port 8000 --discovery-backend file > /dev/null 2>&1 &
python3 -m dynamo.sglang --model-path Qwen/Qwen3-0.6B --discovery-backend file &
curl -s localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "Qwen/Qwen3-0.6B",
"messages": [{"role": "user", "content": "Hello!"}],
"max_tokens": 100
}' | jq
tensorrtllm-runtime:1.3.0
and vllm-runtime:1.3.0
containers are available the same way.
PyPI install #
uv pip install --prerelease=allow "ai-dynamo[sglang]" # or [vllm]
Kubernetes (recommended for production) #
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
name: my-model
spec:
model: Qwen/Qwen3-0.6B
backend: vllm
sla:
ttft: 200.0 # ms
itl: 20.0 # ms
autoApply: true
Pre-built recipes exist for common models (Llama-3-70B on vLLM, DeepSeek-R1 on SGLang disaggregated, Qwen3-32B-FP8 on TensorRT-LLM), plus cloud-specific guides for AWS EKS, Google GKE, Azure AKS, and Amazon ECS.
Two Request-Routing Topologies # #
| Topology | What it is | When to use |
|---|---|---|
| Dynamo-native Frontend routing | Dynamo’s own Frontend + Router make worker-selection decisions, no external gateway needed | Local development, single-cluster deployments |
| Gateway API routing (GAIE) | A Kubernetes Gateway API Inference Extension calls Dynamo’s Endpoint Picker Plugin before forwarding | Platforms standardizing on Gateway API, where policy/auth/rate-limiting should sit at the cluster edge |
Both expose an OpenAI-compatible API and support the same backends, disaggregated serving, and KV-aware routing — the choice is about where routing decisions live in your infrastructure, not a feature trade-off.
Use Cases # #
1. Scaling Past a Single Node #
Once one GPU (or one node) can’t serve your traffic, Dynamo coordinates multiple nodes as one inference system instead of you hand-rolling load balancing across independent engine instances.
2. Cutting Redundant Prefill Compute #
KV-aware routing avoids recomputing prefill for requests that hit workers already holding relevant KV cache — directly behind the published 2× TTFT improvement.
3. Meeting Latency SLAs Without Overprovisioning #
The Planner’s SLA-driven autoscaling targets a specific TTFT/ITL budget rather than scaling on raw CPU/GPU utilization, aimed at the 80% fewer SLA breaches at lower TCO result.
4. Serving Reasoning, Multimodal, or Video Workloads at Scale #
Native support for disaggregated multimodal encode/prefill/decode and video generation (FastVideo, SGLang Diffusion) extends beyond text-only LLM serving.
Related Repositories # #
| Repository | Purpose |
|---|---|
GroveAIConfigurator## Related Articles #
SGLang: Structured Generation for LLMs— one of the three inference engines Dynamo orchestrates rather than replacesSelf-Hosted LLM 2026: Ollama vs vLLM vs LocalAI— for single-node self-hosting, the scale below where Dynamo’s orchestration layer becomes relevant
Conclusion # #
NVIDIA Dynamo targets a specific, well-defined gap: individual inference engines are excellent at serving one model efficiently on one GPU or node, but coordinating dozens of GPUs — disaggregating prefill from decode, routing around redundant KV computation, autoscaling to an SLA rather than a CPU graph — is a separate, harder problem most engines don’t solve themselves. Apache-2.0 licensed, backed by NVIDIA, and unusually disciplined about sourcing its own performance claims, it’s a serious option once you’ve outgrown “one engine, one GPU” — and the README itself tells you plainly when you haven’t.
Best for: Teams running LLM inference across multiple GPUs or nodes who need disaggregated serving, KV-aware routing, or SLA-driven autoscaling on top of vLLM, SGLang, or TensorRT-LLM — not single-GPU deployments.
GitHub: https://github.com/ai-dynamo/dynamo
Last updated: 2026-08-03