llm-d 1.0: The Open-Source Control Plane for vLLM Fleets

Llm-d 1.0, an open-source control plane for vLLM fleets on Kubernetes, shipped on October 20 and is being spotlighted at KubeCon North America this week. The project addresses wasted KV cache in multi-replica deployments by subscribing to each vLLM replica's state so routing can account for what is cached where, rather than round-robining requests through a load balancer. The release matters for teams running LLM inference at scale, where redundant prompt-prefix recomputation across five to ten replicas erodes the benefit of prefix caching.

Most teams running LLMs on Kubernetes are quietly burning money. They deploy five or ten vLLM replicas behind a load balancer and call it production. The load balancer round-robins requests. Each replica recomputes the same prompt prefixes it already processed an hour ago. KV cache — the whole point of prefix caching — gets wasted because the router has no idea what’s cached where. llm-d 1.0, which shipped October 20 and is getting its KubeCon North America spotlight this week, fixes that. It is an open-source control plane that sits in front of a vLLM fleet, subscribes to each replica’s … The post