llm-d 1.0: The Open-Source Control Plane for vLLM Fleets

Most teams running LLMs on Kubernetes are quietly burning money. They deploy five or ten vLLM replicas behind a load balancer and call it production. The load balancer round-robins requests. Each replica recomputes the same prompt prefixes it already processed an hour ago. KV cache — the whole point of prefix caching — gets wasted because the router has no idea what’s cached where. llm-d 1.0, which shipped October 20 and is getting its KubeCon North America spotlight this week, fixes that. It is an open-source control plane that sits in front of a vLLM fleet, subscribes to each replica’s […]

The post