# NVIDIA Dynamo: The Orchestration Layer That Turns vLLM Into a Cluster

> Source: <https://dibi8.com/resources/llm-frameworks/nvidia-dynamo-distributed-inference-2026/>
> Published: 2026-08-02 17:55:00+00:00

# NVIDIA Dynamo: The Orchestration Layer That Turns vLLM Into a Cluster

NVIDIA Dynamo is an Apache-2.0 datacenter-scale inference orchestration layer that sits above vLLM, SGLang, and TensorRT-LLM — disaggregated prefill/decode, KV-aware routing, multi-tier KV caching, and SLA-driven autoscaling, with published benchmarks up to 750x higher throughput on GB300 NVL72.

- ⭐ 7658
- Rust
- Python
- Apache-2.0
- Updated 2026-08-03

[SGLang: Structured Generation for LLMs](https://dibi8.com/resources/llm-frameworks/sglang-structured-generation-llm/) •
[Self-Hosted LLM 2026: Ollama vs vLLM vs LocalAI](https://dibi8.com/resources/llm-frameworks/self-hosted-llm-2026-ollama-vllm-localai/)

*Project banner — from github.com/ai-dynamo/dynamo*

## What Is NVIDIA Dynamo? [#](#what-is-nvidia-dynamo)

**Dynamo** is NVIDIA’s open-source, datacenter-scale inference stack — but the framing that matters most is what it *isn’t*: it doesn’t replace SGLang, TensorRT-LLM, or vLLM. It’s **the orchestration layer above them**, turning a cluster of GPUs running any of those engines into a coordinated multi-node inference system, with disaggregated serving, intelligent routing, multi-tier KV caching, and automatic scaling working together.

🔗 **GitHub**: [https://github.com/ai-dynamo/dynamo](https://github.com/ai-dynamo/dynamo)
📖 **Docs**: [docs.nvidia.com/dynamo](https://docs.nvidia.com/dynamo/)

Apache-2.0 licensed, built in **Rust for performance, Python for extensibility**, at **7,600+ GitHub stars** with **160+ community contributors**, and a commit from **August 1, 2026** — the day before this article.

## When to Use It (and When Not To) [#](#when-to-use-it-and-when-not-to)

The README is upfront about the boundary:

- You’re serving LLMs across
**multiple GPUs or nodes** and need to coordinate them - You want
**KV-aware routing** to avoid redundant prefill computation - You need to
**independently scale prefill and decode**(disaggregated serving) - You want
**automatic scaling** that meets latency SLAs at minimum total cost of ownership - You need
**fast cold-starts** when spinning up new replicas

**“If you’re running a single model on a single GPU, your inference engine alone is probably sufficient.”** — that’s not a hedge, it’s the project telling you when not to add the complexity.

## Backend Support [#](#backend-support)

| Feature | SGLang | TensorRT-LLM | vLLM |
|---|---|---|---|
| Disaggregated Serving | ✅ | ✅ | ✅ |
| KV-Aware Routing | ✅ | ✅ | ✅ |
| SLA-Based Planner | ✅ | ✅ | ✅ |
| KVBM (multi-tier KV cache) | 🚧 | ✅ | ✅ |
| Multimodal | ✅ | ✅ | ✅ |
| Tool Calling | ✅ | ✅ | ✅ |

## Published Results (Each Sourced, Not Just Asserted) [#](#published-results-each-sourced-not-just-asserted)

| Result | Context |
|---|---|
7× higher throughput per GPU | DeepSeek R1 on GB200 NVL72 with Dynamo vs. B200 without (per
|

**7×** faster model startup**2×** faster time to first token[Baseten benchmark](https://www.baseten.co/blog/how-baseten-achieved-2x-faster-inference-with-nvidia-dynamo/))**80%** fewer SLA breaches**750×** higher throughputEach of these is attributed to a specific benchmark or source rather than presented as an unsourced headline number — worth checking the linked methodology before treating any single figure as your expected result.

## Core Capabilities [#](#core-capabilities)

| Capability | What it does | Why it matters |
|---|---|---|
Disaggregated Prefill/Decode | Separates prefill and decode into independently scalable GPU pools | Each phase runs on hardware tuned for its workload |
KV-Aware Routing | Routes requests based on worker load and KV cache overlap | Eliminates redundant prefill — the 2× TTFT improvement above |
KV Block Manager (KVBM) | Offloads KV cache across GPU → CPU → SSD → remote storage | Extends effective context length beyond raw GPU memory |
ModelExpress | Streams model weights GPU-to-GPU via NIXL/NVLink | 7× faster cold-start for new replicas |
Planner | SLA-driven autoscaler that profiles workloads and right-sizes pools | Meets latency targets at minimum TCO |
Grove | Kubernetes operator for topology-aware gang scheduling (NVL72) | Places workloads optimally across racks, hosts, NUMA nodes |
AIConfigurator | Simulates 10,000+ deployment configs in seconds | Finds an optimal serving config without burning GPU-hours on trial and error |
Fault Tolerance | Canary health checks + in-flight request migration | Workers can fail without failing user requests |

### New in 1.0 [#](#new-in-10)

**Zero-config deploy (DGDR, beta)**— specify model, hardware, and SLA in one YAML; AIConfigurator profiles the workload and Planner optimizes topology automatically**Agentic inference**— per-request hints for priority, expected output length, speculative prefill, plus LangChain and NeMo Agent Toolkit integrations**Multimodal E/P/D**— disaggregated encode/prefill/decode with embedding cache, 30% faster TTFT on image workloads** Video generation**— native FastVideo + SGLang Diffusion support, real-time 1080p on a single B200** K8s Inference Gateway plugin**— KV-aware routing inside the standard Kubernetes gateway

## Quick Start [#](#quick-start)

### Container (fastest) [#](#container-fastest)

```
docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.3.0
# Inside the container — start frontend and worker
python3 -m dynamo.frontend --http-port 8000 --discovery-backend file > /dev/null 2>&1 &
python3 -m dynamo.sglang --model-path Qwen/Qwen3-0.6B --discovery-backend file &
curl -s localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-0.6B",
  "messages": [{"role": "user", "content": "Hello!"}],
  "max_tokens": 100
}' | jq
```

`tensorrtllm-runtime:1.3.0`

and `vllm-runtime:1.3.0`

containers are available the same way.

### PyPI install [#](#pypi-install)

```
uv pip install --prerelease=allow "ai-dynamo[sglang]"   # or [vllm]
```

### Kubernetes (recommended for production) [#](#kubernetes-recommended-for-production)

```
# Zero-config deploy: specify model + SLA, Dynamo handles the rest
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: my-model
spec:
  model: Qwen/Qwen3-0.6B
  backend: vllm
  sla:
    ttft: 200.0   # ms
    itl: 20.0     # ms
  autoApply: true
```

Pre-built recipes exist for common models (Llama-3-70B on vLLM, DeepSeek-R1 on SGLang disaggregated, Qwen3-32B-FP8 on TensorRT-LLM), plus cloud-specific guides for AWS EKS, Google GKE, Azure AKS, and Amazon ECS.

## Two Request-Routing Topologies [#](#two-request-routing-topologies)

| Topology | What it is | When to use |
|---|---|---|
Dynamo-native Frontend routing | Dynamo’s own Frontend + Router make worker-selection decisions, no external gateway needed | Local development, single-cluster deployments |
Gateway API routing (GAIE) | A Kubernetes Gateway API Inference Extension calls Dynamo’s Endpoint Picker Plugin before forwarding | Platforms standardizing on Gateway API, where policy/auth/rate-limiting should sit at the cluster edge |

Both expose an OpenAI-compatible API and support the same backends, disaggregated serving, and KV-aware routing — the choice is about where routing decisions live in your infrastructure, not a feature trade-off.

## Use Cases [#](#use-cases)

### 1. Scaling Past a Single Node [#](#1-scaling-past-a-single-node)

Once one GPU (or one node) can’t serve your traffic, Dynamo coordinates multiple nodes as one inference system instead of you hand-rolling load balancing across independent engine instances.

### 2. Cutting Redundant Prefill Compute [#](#2-cutting-redundant-prefill-compute)

KV-aware routing avoids recomputing prefill for requests that hit workers already holding relevant KV cache — directly behind the published 2× TTFT improvement.

### 3. Meeting Latency SLAs Without Overprovisioning [#](#3-meeting-latency-slas-without-overprovisioning)

The Planner’s SLA-driven autoscaling targets a specific TTFT/ITL budget rather than scaling on raw CPU/GPU utilization, aimed at the 80% fewer SLA breaches at lower TCO result.

### 4. Serving Reasoning, Multimodal, or Video Workloads at Scale [#](#4-serving-reasoning-multimodal-or-video-workloads-at-scale)

Native support for disaggregated multimodal encode/prefill/decode and video generation (FastVideo, SGLang Diffusion) extends beyond text-only LLM serving.

## Related Repositories [#](#related-repositories)

| Repository | Purpose |
|---|---|
|

[Grove](https://github.com/ai-dynamo/grove)[AIConfigurator](https://github.com/ai-dynamo/aiconfigurator)## Related Articles [#](#related-articles)

[SGLang: Structured Generation for LLMs](https://dibi8.com/resources/llm-frameworks/sglang-structured-generation-llm/)— one of the three inference engines Dynamo orchestrates rather than replaces[Self-Hosted LLM 2026: Ollama vs vLLM vs LocalAI](https://dibi8.com/resources/llm-frameworks/self-hosted-llm-2026-ollama-vllm-localai/)— for single-node self-hosting, the scale below where Dynamo’s orchestration layer becomes relevant

## Conclusion [#](#conclusion)

**NVIDIA Dynamo** targets a specific, well-defined gap: individual inference engines are excellent at serving one model efficiently on one GPU or node, but coordinating dozens of GPUs — disaggregating prefill from decode, routing around redundant KV computation, autoscaling to an SLA rather than a CPU graph — is a separate, harder problem most engines don’t solve themselves. Apache-2.0 licensed, backed by NVIDIA, and unusually disciplined about sourcing its own performance claims, it’s a serious option once you’ve outgrown “one engine, one GPU” — and the README itself tells you plainly when you haven’t.

**Best for**: Teams running LLM inference across multiple GPUs or nodes who need disaggregated serving, KV-aware routing, or SLA-driven autoscaling on top of vLLM, SGLang, or TensorRT-LLM — not single-GPU deployments.

**GitHub**: [https://github.com/ai-dynamo/dynamo](https://github.com/ai-dynamo/dynamo)

*Last updated: 2026-08-03*