{"slug": "modelplane-v0-4-nvidia-dynamo-and-ai-cluster-runtime", "title": "Modelplane v0.4: NVIDIA Dynamo and AI Cluster Runtime", "summary": "Modelplane released version 0.4, adding an NVIDIA Dynamo serving stack that brings gang scheduling and peer-to-peer weight transfer to its inference clusters. The release, built with NVIDIA's Dynamo team, also pulls serving-stack versions and configuration from NVIDIA AI Cluster Runtime, which validates them on real hardware. Modelplane now starts a cloud provider only when a cluster is provisioned on that cloud, and the serving-stack choice is immutable so platform teams can migrate deployments cluster by cluster.", "body_md": "Modelplane v0.4 is out, and its two headline features both came out of work with\nNVIDIA. A new `Dynamo` serving stack brings gang scheduling and peer-to-peer\nweight transfer to a cluster. Modelplane also now takes the versions and\nconfiguration for much of each cloud's serving stack from NVIDIA AI Cluster\nRuntime, which validates them on real hardware. Modelplane composes NVIDIA's\nstack rather than reimplementing it.\n\nWe've also made Modelplane a little more efficient. It now only starts a provider when you provision a cluster on that cloud. Here's what's new.\n\n## NVIDIA Dynamo, fleet-wide[#](#nvidia-dynamo-fleet-wide)\n\nWe designed the new `Dynamo` serving stack with NVIDIA's\n[Dynamo](https://developer.nvidia.com/dynamo) team. You can now choose the\nserving stack each Modelplane-managed cluster runs.\n[Grove](https://github.com/ai-dynamo/grove) and the [KAI\nScheduler](https://github.com/NVIDIA/KAI-Scheduler) place a multi-node engine as\na gang, and [ModelExpress](https://github.com/ai-dynamo/modelexpress) moves\nmodel weights GPU to GPU between replicas.\n\nThe `ModelDeployment` an ML team writes is unchanged. The same manifest runs\non either stack. Which stack a cluster runs is a platform decision, made per\ncluster, so a fleet can run both at once.\n\n## The cluster serving stack[#](#the-cluster-serving-stack)\n\nModelplane operates a fleet. It provisions inference clusters and node pools, schedules model replicas onto hardware that fits, and fronts the whole fleet with an OpenAI-compatible endpoint. It isn't a serving layer itself.\n\nA serving stack owns what happens inside one cluster. It places a multi-node engine's pods and gets the model's weights into GPU memory. Dynamo does both, and it does things Modelplane's current \"standard\" stack doesn't, like gang scheduling, P2P weight transfer, and keeping weights resident in GPU memory across an engine crash.\n\n## Opting a cluster in[#](#opting-a-cluster-in)\n\nOn a `Dynamo` cluster Modelplane installs Grove, the KAI Scheduler, and the\nModelExpress server. On a `Standard` cluster it installs the\n[LeaderWorkerSet](https://github.com/kubernetes-sigs/lws) controller. Everything\nelse about a cluster, from how it fronts requests to how it stages model\nweights, is the same on both.\n\nThe choice is immutable, so adoption is incremental. A platform team stands up a Dynamo cluster next to the ones it already runs and moves deployments over cluster by cluster.\n\n## Gang scheduling with Grove and the KAI Scheduler[#](#gang-scheduling-with-grove-and-the-kai-scheduler)\n\nGang scheduling makes the most of the GPU time you're paying for. A multi-node engine is a gang. Its leader and workers are useless apart. Schedule those pods one at a time and a gang can half-land, holding GPUs while serving nothing, waiting for nodes that may not be free for a while. KAI places the whole gang or none of it.\n\nModelplane composes an engine onto whichever stack its cluster runs. A `Leader`\nand `Worker` gang is a LeaderWorkerSet on `Standard`, and on `Dynamo` a Grove\n`PodCliqueSet` with a leader clique and a worker clique, scheduled by KAI.\n\nSo a serving stack has to run two pod specs with distinct commands, and give a worker a way to find its leader. Grove does both.\n\n## Weight transfer with ModelExpress[#](#weight-transfer-with-modelexpress)\n\nLoading weights is slow. Each replica reads the model from storage before it can serve a token, and several replicas scaling up together compete for reads from the same storage. ModelExpress makes that one read rather than one per replica.\n\nA `Dynamo` cluster runs a ModelExpress server. The server remembers and\nadvertises which replica holds a model in GPU memory. The first replica loads\nfrom the cache volume and publishes itself as a source, and later replicas pull\nthe weights from a peer's GPU over RDMA, across a fast fabric like EFA on EKS. A\nreplica that finds no peer, or no fabric to reach one over, reads the cache\nvolume instead, so size and keep the cache for every replica on either stack.\n\n## The same manifest on either stack[#](#the-same-manifest-on-either-stack)\n\nThe ML team authors a `ModelDeployment` the same way they always have. A\n`ModelDeployment` describes the inference engines Modelplane should run. It says\nnothing about the stack underneath it.\n\nHere's a 480B model across two nodes, tensor-parallel within each node and\npipeline-parallel across them, that also opts into ModelExpress.\n`$(MODELPLANE_LEADER_ADDRESS)` is the address the leader is reachable at, and it\nresolves on both stacks:\n\nThe `modelCacheRef` names a `ModelCache`, which stages a model's weights once\nper cluster on shared storage. The `--load-format modelexpress` flag configures\nvLLM to attempt to load weights via ModelExpress. Run it on a `Standard`\ncluster, where nothing runs a ModelExpress server, and the engine reads the\ncached weights from a persistent volume just like it does today.\n\n`$(MODELPLANE_LEADER_ADDRESS)` resolves on Dynamo because the Dynamo team\nchanged Grove so it could. Grove injected its pod-discovery variables in an\norder that left a pod template unable to derive a value from them. We hit it\nwhile building the stack, and Stefan from the Dynamo team fixed it in\n[grove#753](https://github.com/ai-dynamo/grove/pull/753).\n\n## Serving stacks, built with NVIDIA AI Cluster Runtime[#](#serving-stacks-built-with-nvidia-ai-cluster-runtime)\n\nA serving stack is more than the engine that serves tokens. Under it sits a layer of GPU-adjacent software: GPU and DRA drivers, GPU and network operators, and a scheduler. All of it has to agree with the GPU, the node OS, and the Kubernetes release beneath it. Those versions interact, and a combination that's wrong for the hardware causes failures and slowdowns that are hard to diagnose. Finding a known-good set for a cloud, GPU, and OS is real work. NVIDIA used to do it on its own hardware, in-house.\n\nNVIDIA now publishes that work. [NVIDIA AI Cluster\nRuntime](https://github.com/NVIDIA/aicr) (AICR) is an Apache 2.0 tool that\ncaptures known-good combinations of drivers, operators, and configuration and\npublishes them as version-locked recipes, one per cloud, accelerator, and OS,\nvalidated on the hardware by NVIDIA and the AICR community.\n\nIn v0.4 Modelplane builds each cloud's stack from a fixed list of components. Much of it comes from AICR: the GPU drivers and operators, node tuning, and cluster monitoring. Modelplane supplies the serving pieces on top: the gateway, weight transfer, and the leader and worker controllers.\n\nA release now installs one stack per cloud, the same on every cluster, tested as a whole. A cluster's spec no longer carries version fields: nothing to tune, and nothing to get wrong.\n\n## Providers that start on demand[#](#providers-that-start-on-demand)\n\nA Modelplane control plane can provision clusters on any cloud it supports, but\na given one usually runs one or two. v0.4 stops it running every cloud's\nproviders regardless. A control plane now starts a cloud's providers only once\nit has an `InferenceCluster` on that cloud, and leaves the rest idle.\n\n## What's next with NVIDIA[#](#whats-next-with-nvidia)\n\nWe're not done building on NVIDIA's stack. Work with the Dynamo team, some already in flight and some still ahead, would let Modelplane compose more of it.\n\nThe Dynamo team is working on\n[grove#755](https://github.com/ai-dynamo/grove/pull/755), which exposes a\nscaling-group-wide pod index. That would let us alias a rank the way we alias\nthe leader address, so every pod of a gang could share one command.\n\nFurther out, we plan to power the `Dynamo` stack with Dynamo's\n`DynamoGraphDeployment` (DGD) custom resource, which drives Grove, KAI,\nModelExpress, and the GPU Memory Service (GMS). That depends on changes from the\nDynamo team: distinct leader and worker pod specs the operator won't rewrite\n([dynamo#12696](https://github.com/ai-dynamo/dynamo/issues/12696)), and serving\nfrom a stock engine image\n([dynamo#10835](https://github.com/ai-dynamo/dynamo/issues/10835)).\n\nEvery layer NVIDIA builds inside a cluster is a layer Modelplane can compose across a fleet.\n\n## Try it[#](#try-it)\n\nThe full release notes are on\n[GitHub](https://github.com/modelplaneai/modelplane/releases/tag/v0.4.0). The\n[getting-started guide](https://docs.modelplane.ai/getting-started/) covers\nstanding up a fleet, and\n[how it works](https://docs.modelplane.ai/overview/how-it-works/) covers what a\nserving stack installs. Modelplane is Apache 2.0 and moving fast at\n[github.com/modelplaneai/modelplane](https://github.com/modelplaneai/modelplane),\nand questions are welcome in [Slack](https://slack.modelplane.ai).", "url": "https://wpnews.pro/news/modelplane-v0-4-nvidia-dynamo-and-ai-cluster-runtime", "canonical_source": "https://modelplane.ai/blog/modelplane-v0-4", "published_at": "2026-09-02 00:00:00+00:00", "updated_at": "2026-09-10 18:07:16.956750+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "mlops", "developer-tools"], "entities": ["Modelplane", "NVIDIA", "NVIDIA Dynamo", "NVIDIA AI Cluster Runtime", "Grove", "KAI Scheduler", "ModelExpress", "LeaderWorkerSet"], "alternates": {"html": "https://wpnews.pro/news/modelplane-v0-4-nvidia-dynamo-and-ai-cluster-runtime", "markdown": "https://wpnews.pro/news/modelplane-v0-4-nvidia-dynamo-and-ai-cluster-runtime.md", "text": "https://wpnews.pro/news/modelplane-v0-4-nvidia-dynamo-and-ai-cluster-runtime.txt", "jsonld": "https://wpnews.pro/news/modelplane-v0-4-nvidia-dynamo-and-ai-cluster-runtime.jsonld"}}