{"slug": "modelplane-v0-5-an-ai-gateway-and-fleet-telemetry", "title": "Modelplane v0.5: an AI gateway and fleet telemetry", "summary": "Modelplane released version 0.5, turning its fleet gateway into an AI gateway built on Agent Router that authenticates callers, reads the model named in each request body, and fails over between backends using new priority settings alongside weight. The release adds an OpenTelemetry collector on every inference cluster that renames engine-specific series such as vLLM's `vllm:num_requests_waiting` and SGLang's `sglang:num_queue_reqs` into a single `modelplane_*` vocabulary, with `TelemetryDestination` and `MetricMapping` resources controlling where metrics go and how units convert. Modelplane 0.5 also lets a `ModelDeployment` scale to zero replicas and adds Civo to the clouds it can provision an `InferenceCluster` on, joining EKS, AKS, GKE, Nebius and Vultr.", "body_md": "Modelplane v0.5 is out. The fleet gateway is now an AI gateway. It authenticates callers and reads the model a request asks for, then fails over between the backends that serve it. Modelplane also collects your fleet's metrics under one set of names, whatever engine produced them. And Civo joins the clouds Modelplane can provision a cluster on.\n\nA `ModelDeployment` can also scale to zero replicas now, so one you aren't\nserving from costs you no GPUs. Here's what's new.\n\n## The fleet gateway is an AI gateway[#](#the-fleet-gateway-is-an-ai-gateway)\n\nThe fleet gateway used to be an HTTP router on the control plane. It understood\nnothing about the requests it forwarded. A caller reached a `ModelService` by\npath prefix, nothing authenticated them, and the hop out to each cluster\ncrossed the public internet in plain HTTP.\n\nIt's now [Agent Router](https://theagentrouter.ai/), running on an\n`InferenceCluster` you name rather than on the control plane, and the same one\nevery cluster already runs at its edge. We run a build from before the\nproject left the Envoy family and took that name, so you'll still see\n`envoy-ai-gateway-system` on a cluster. It reads the model a request\nnames in its body and resolves the `ModelService` that serves it. For the\nbackend it picks, it rewrites the model name, the credential and the path, so a\nbackend sees the name it knows and a caller's key never reaches a third party.\n\n`ModelService` gains `priority` alongside `weight`. Weight splits traffic\nbetween backends at one priority; priority fails over to the next when they go\nunhealthy. Every request meters a token count per caller, streams included.\n\nYou can now run more than one `InferenceGateway`. Each names the cluster it\nruns on, so a fleet can have one per region for residency, or two in a region\nfor availability:\n\nTraffic from a fleet gateway to a cluster gateway is now mutually authenticated and encrypted. It used to cross the public internet as plain HTTP.\n\n## One vocabulary for a fleet's metrics[#](#one-vocabulary-for-a-fleets-metrics)\n\nModelplane doesn't own your engine. You bring the image and the command, and a\n`ModelDeployment` runs vLLM, SGLang, or anything else that speaks the OpenAI\nAPI, without Modelplane knowing anything about it.\n\nThat freedom is what makes a fleet hard to watch. vLLM publishes\n`vllm:num_requests_waiting`. SGLang calls the same measurement\n`sglang:num_queue_reqs`. [DCGM](https://developer.nvidia.com/dcgm), NVIDIA's\nGPU exporter, reports framebuffer memory in mebibytes under a name that says\nbytes, and energy in millijoules under a name that says joules.\nA dashboard written against one engine is wrong on the next, and a fleet\nrunning both has no fleet-wide number at all. Standardising on one engine would\ncost you the choice that made Modelplane worth using, so Modelplane normalises\nthe names where it collects them instead.\n\nModelplane now runs an OpenTelemetry collector on every inference cluster. It\ndiscovers every component Modelplane installs, renames each one's series into a\nsingle `modelplane_*` vocabulary, and exports them wherever you say.\n\nTwo new kinds. A `TelemetryDestination` says where metrics go, and nothing is\ncollected until one exists:\n\nA `MetricMapping` says what a component emits and what Modelplane calls it.\nModelplane comes with mappings for vLLM, SGLang, the gateway, the endpoint\npicker and DCGM, so those need nothing from you. Write one for an engine Modelplane\nhas never seen and its numbers join the same surface:\n\n`fromUnit` is the one to get right. A metric's name is no guide to its unit, so\nthe mapping says what the source measures in and Modelplane converts, histogram\nbuckets and all. Skip it and a series named `_seconds` holding milliseconds\nreads a thousand times fast, with nothing downstream to catch it.\n\n## Civo[#](#civo)\n\n[Civo](https://www.civo.com/) joins EKS, AKS, GKE, Nebius and Vultr as a cloud\nModelplane can provision an `InferenceCluster` on, with the same spec you'd\nwrite for any of them.\n\nTwo things about Civo needed handling underneath. Its GPU images carry no\nNVIDIA driver, so the serving stack installs the GPU Operator to supply one,\nwith the toolkit and device plugin switched off so the DRA driver stays the\nonly thing allocating GPUs. And Civo has no server-side autoscaler, so a pool\nwith a `maxNodeCount` is scaled by the upstream cluster-autoscaler running on\nthe cluster itself. Civo's volumes are ReadWriteOnce, so `ModelCache` isn't\navailable there yet.\n\n## Scaling to zero[#](#scaling-to-zero)\n\nA `ModelDeployment` can now scale to zero replicas. `spec.replicas` used to carry\na floor of one, so `kubectl scale --replicas=0` was rejected at admission. That\nis awkward when scaling to zero is most of the reason to put KEDA in front of a\nGPU workload in the first place. There was no way to park a deployment\neither: withdrawing its endpoints while keeping the object meant tainting the\ncluster hosting it.\n\nDropping the floor on its own would have made a parked deployment look broken. Zero desired replicas against an empty schedule reads as none of them scheduled, and on a control plane with no clusters it reads as having nowhere to run, so a deployment you had deliberately parked would sit there permanently not ready.\n\nZero now takes a path of its own. Nothing is composed, so the deployment's\n`ModelReplicas` and `ModelEndpoints` are pruned, `status.replicas` reports 0 to\nthe scale subresource, and readiness reports true with a `ScaledToZero` reason,\nthe way a Deployment at zero replicas still reports Available. A parked\ndeployment reads as parked rather than as failing, which is what makes it safe\nfor an autoscaler to do on your behalf.\n\n## Try it[#](#try-it)\n\nThe [getting-started guide](https://docs.modelplane.ai/getting-started/) covers\nstanding up a fleet, and\n[Monitor the Fleet](https://docs.modelplane.ai/platform/telemetry/) covers\npointing telemetry at a backend you already run. Modelplane is Apache 2.0 and\nmoving fast at\n[github.com/modelplaneai/modelplane](https://github.com/modelplaneai/modelplane),\nand questions are welcome in [Slack](https://slack.modelplane.ai).", "url": "https://wpnews.pro/news/modelplane-v0-5-an-ai-gateway-and-fleet-telemetry", "canonical_source": "https://modelplane.ai/blog/modelplane-v0-5", "published_at": "2026-10-30 00:00:00+00:00", "updated_at": "2026-11-03 21:53:00.724061+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "ai-tools", "ai-agents"], "entities": ["Modelplane", "Agent Router", "Civo", "vLLM", "SGLang", "NVIDIA", "DCGM", "Envoy"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/modelplane-v0-5-an-ai-gateway-and-fleet-telemetry", "markdown": "https://wpnews.pro/news/modelplane-v0-5-an-ai-gateway-and-fleet-telemetry.md", "text": "https://wpnews.pro/news/modelplane-v0-5-an-ai-gateway-and-fleet-telemetry.txt", "jsonld": "https://wpnews.pro/news/modelplane-v0-5-an-ai-gateway-and-fleet-telemetry.jsonld"}}