Modelplane v0.4: NVIDIA Dynamo and AI Cluster Runtime

Modelplane released version 0.4, adding an NVIDIA Dynamo serving stack that brings gang scheduling and peer-to-peer weight transfer to its inference clusters. The release, built with NVIDIA's Dynamo team, also pulls serving-stack versions and configuration from NVIDIA AI Cluster Runtime, which validates them on real hardware. Modelplane now starts a cloud provider only when a cluster is provisioned on that cloud, and the serving-stack choice is immutable so platform teams can migrate deployments cluster by cluster.

Modelplane v0.4 is out, and its two headline features both came out of work with NVIDIA. A new Dynamo serving stack brings gang scheduling and peer-to-peer weight transfer to a cluster. Modelplane also now takes the versions and configuration for much of each cloud's serving stack from NVIDIA AI Cluster Runtime, which validates them on real hardware. Modelplane composes NVIDIA's stack rather than reimplementing it. We've also made Modelplane a little more efficient. It now only starts a provider when you provision a cluster on that cloud. Here's what's new. NVIDIA Dynamo, fleet-wide nvidia-dynamo-fleet-wide We designed the new Dynamo serving stack with NVIDIA's Dynamo https://developer.nvidia.com/dynamo team. You can now choose the serving stack each Modelplane-managed cluster runs. Grove https://github.com/ai-dynamo/grove and the KAI Scheduler https://github.com/NVIDIA/KAI-Scheduler place a multi-node engine as a gang, and ModelExpress https://github.com/ai-dynamo/modelexpress moves model weights GPU to GPU between replicas. The ModelDeployment an ML team writes is unchanged. The same manifest runs on either stack. Which stack a cluster runs is a platform decision, made per cluster, so a fleet can run both at once. The cluster serving stack the-cluster-serving-stack Modelplane operates a fleet. It provisions inference clusters and node pools, schedules model replicas onto hardware that fits, and fronts the whole fleet with an OpenAI-compatible endpoint. It isn't a serving layer itself. A serving stack owns what happens inside one cluster. It places a multi-node engine's pods and gets the model's weights into GPU memory. Dynamo does both, and it does things Modelplane's current "standard" stack doesn't, like gang scheduling, P2P weight transfer, and keeping weights resident in GPU memory across an engine crash. Opting a cluster in opting-a-cluster-in On a Dynamo cluster Modelplane installs Grove, the KAI Scheduler, and the ModelExpress server. On a Standard cluster it installs the LeaderWorkerSet https://github.com/kubernetes-sigs/lws controller. Everything else about a cluster, from how it fronts requests to how it stages model weights, is the same on both. The choice is immutable, so adoption is incremental. A platform team stands up a Dynamo cluster next to the ones it already runs and moves deployments over cluster by cluster. Gang scheduling with Grove and the KAI Scheduler gang-scheduling-with-grove-and-the-kai-scheduler Gang scheduling makes the most of the GPU time you're paying for. A multi-node engine is a gang. Its leader and workers are useless apart. Schedule those pods one at a time and a gang can half-land, holding GPUs while serving nothing, waiting for nodes that may not be free for a while. KAI places the whole gang or none of it. Modelplane composes an engine onto whichever stack its cluster runs. A Leader and Worker gang is a LeaderWorkerSet on Standard , and on Dynamo a Grove PodCliqueSet with a leader clique and a worker clique, scheduled by KAI. So a serving stack has to run two pod specs with distinct commands, and give a worker a way to find its leader. Grove does both. Weight transfer with ModelExpress weight-transfer-with-modelexpress Loading weights is slow. Each replica reads the model from storage before it can serve a token, and several replicas scaling up together compete for reads from the same storage. ModelExpress makes that one read rather than one per replica. A Dynamo cluster runs a ModelExpress server. The server remembers and advertises which replica holds a model in GPU memory. The first replica loads from the cache volume and publishes itself as a source, and later replicas pull the weights from a peer's GPU over RDMA, across a fast fabric like EFA on EKS. A replica that finds no peer, or no fabric to reach one over, reads the cache volume instead, so size and keep the cache for every replica on either stack. The same manifest on either stack the-same-manifest-on-either-stack The ML team authors a ModelDeployment the same way they always have. A ModelDeployment describes the inference engines Modelplane should run. It says nothing about the stack underneath it. Here's a 480B model across two nodes, tensor-parallel within each node and pipeline-parallel across them, that also opts into ModelExpress. $ MODELPLANE LEADER ADDRESS is the address the leader is reachable at, and it resolves on both stacks: The modelCacheRef names a ModelCache , which stages a model's weights once per cluster on shared storage. The --load-format modelexpress flag configures vLLM to attempt to load weights via ModelExpress. Run it on a Standard cluster, where nothing runs a ModelExpress server, and the engine reads the cached weights from a persistent volume just like it does today. $ MODELPLANE LEADER ADDRESS resolves on Dynamo because the Dynamo team changed Grove so it could. Grove injected its pod-discovery variables in an order that left a pod template unable to derive a value from them. We hit it while building the stack, and Stefan from the Dynamo team fixed it in grove 753 https://github.com/ai-dynamo/grove/pull/753 . Serving stacks, built with NVIDIA AI Cluster Runtime serving-stacks-built-with-nvidia-ai-cluster-runtime A serving stack is more than the engine that serves tokens. Under it sits a layer of GPU-adjacent software: GPU and DRA drivers, GPU and network operators, and a scheduler. All of it has to agree with the GPU, the node OS, and the Kubernetes release beneath it. Those versions interact, and a combination that's wrong for the hardware causes failures and slowdowns that are hard to diagnose. Finding a known-good set for a cloud, GPU, and OS is real work. NVIDIA used to do it on its own hardware, in-house. NVIDIA now publishes that work. NVIDIA AI Cluster Runtime https://github.com/NVIDIA/aicr AICR is an Apache 2.0 tool that captures known-good combinations of drivers, operators, and configuration and publishes them as version-locked recipes, one per cloud, accelerator, and OS, validated on the hardware by NVIDIA and the AICR community. In v0.4 Modelplane builds each cloud's stack from a fixed list of components. Much of it comes from AICR: the GPU drivers and operators, node tuning, and cluster monitoring. Modelplane supplies the serving pieces on top: the gateway, weight transfer, and the leader and worker controllers. A release now installs one stack per cloud, the same on every cluster, tested as a whole. A cluster's spec no longer carries version fields: nothing to tune, and nothing to get wrong. Providers that start on demand providers-that-start-on-demand A Modelplane control plane can provision clusters on any cloud it supports, but a given one usually runs one or two. v0.4 stops it running every cloud's providers regardless. A control plane now starts a cloud's providers only once it has an InferenceCluster on that cloud, and leaves the rest idle. What's next with NVIDIA whats-next-with-nvidia We're not done building on NVIDIA's stack. Work with the Dynamo team, some already in flight and some still ahead, would let Modelplane compose more of it. The Dynamo team is working on grove 755 https://github.com/ai-dynamo/grove/pull/755 , which exposes a scaling-group-wide pod index. That would let us alias a rank the way we alias the leader address, so every pod of a gang could share one command. Further out, we plan to power the Dynamo stack with Dynamo's DynamoGraphDeployment DGD custom resource, which drives Grove, KAI, ModelExpress, and the GPU Memory Service GMS . That depends on changes from the Dynamo team: distinct leader and worker pod specs the operator won't rewrite dynamo 12696 https://github.com/ai-dynamo/dynamo/issues/12696 , and serving from a stock engine image dynamo 10835 https://github.com/ai-dynamo/dynamo/issues/10835 . Every layer NVIDIA builds inside a cluster is a layer Modelplane can compose across a fleet. Try it try-it The full release notes are on GitHub https://github.com/modelplaneai/modelplane/releases/tag/v0.4.0 . The getting-started guide https://docs.modelplane.ai/getting-started/ covers standing up a fleet, and how it works https://docs.modelplane.ai/overview/how-it-works/ covers what a serving stack installs. Modelplane is Apache 2.0 and moving fast at github.com/modelplaneai/modelplane https://github.com/modelplaneai/modelplane , and questions are welcome in Slack https://slack.modelplane.ai .