# Kubernetes for LLM Inference: How AI Workloads Run Across a GPU Cluster > Source: > Published: 2026-09-09 21:01:01+00:00 **TL; DR** Kubernetes does not make inference faster. It runs, places, monitors, replaces and scales the servers that do — and publishes a trustworthy list of which ones are ready. GPUs are allocated to Pods as **whole, indivisible units** through vendor device plugins. This makes GPU scheduling behave very differently from CPU scheduling. A Pod cannot span machines, so a single-node model replica is one Pod — but a replica too large for one machine becomes a group of Pods that must be created and replaced together. An inference Pod takes **minutes** to become ready, because it must pull a large image and load model weights into VRAM. Nearly every operational difference follows from this. **Readiness probes** are what gate traffic, and they are the mechanism that keeps requests away from a server still loading. When a Pod dies, its KV cache and prefix cache die with it. The replacement is correct but cold. Kubernetes autoscaling on CPU is meaningless here; the useful signals come from the inference server itself. A router that makes good decisions needs an accurate list of inference servers: which exist, which are healthy, which are ready to receive traffic. In production that list is never stable. Replicas are added under load and removed when it drops. Servers crash. Deployments roll out new model versions. Hardware fails and nodes are drained. Something has to own that churn — create the servers, place them on machines with the right GPUs, watch their health, replace the dead ones, and publish an accurate picture of what is running. That is orchestration, and in practice it means **Kubernetes**. This article covers what it actually does for an inference deployment, and — more usefully — the specific places where GPU and LLM workloads behave unlike the web services Kubernetes was designed around. Kubernetes is a system for running containerised workloads across a cluster of machines. You describe the state you want — five replicas of this server, each with four GPUs, reachable at this address — and it works continuously to make reality match that description. The important clarification first, because it is the most common misconception: Kubernetes does not participate in inference. It never touches a token, a batch or a KV cache. It starts the process that does, gives it hardware, and gets out of the way. Your throughput comes from the inference engine. Your memory efficiency comes from the engine’s cache management. Kubernetes contributes none of that. What it contributes is everything around it: placement, health, replacement, scaling, networking and rollout. If you run one model on one server, you do not need it. The value appears when you have many GPUs, several models, replicas that fail, and traffic that changes. Five terms are enough to follow the rest. **Cluster** — a set of machines managed together, plus a **control plane** that holds the desired state and drives the cluster toward it. When we say “Kubernetes decides”, we mean the control plane. **Node** — one machine. Some have GPUs, most do not. **Container** — your inference server and its dependencies packaged into an image. **Pod** — the smallest thing Kubernetes schedules. One or more containers sharing a network identity and lifecycle, placed on a single node. For inference, a Pod is typically one container running vLLM or a similar engine. A Pod is not a server you maintain; it is a disposable instance of one. **Deployment** — a declaration of how many identical Pods should exist. Ask for five, and the control plane maintains five, replacing any that die. Two more appear later: **Service**, which gives a stable address over a changing set of Pods, and **probes**, which report whether a Pod is healthy and ready. This is the first genuine difference from ordinary workloads. Kubernetes understands CPU and memory natively, and it treats them as **divisible**. You can request half a CPU core. Two Pods can share a core, and the kernel will time-slice between them. Kubernetes knows nothing about GPUs out of the box. They are exposed by a **device plugin** — software from the hardware vendor that runs on each GPU node, discovers the devices and advertises them to the cluster as a schedulable resource. A Pod then requests them the way it requests memory: ``` resources: limits: nvidia.com/gpu: 4 ``` But GPUs are advertised as **whole, indivisible units**. You cannot request half a GPU by default. If a Pod asks for one, it gets exclusive use of one; no second Pod is placed on it. This has a direct consequence people run into immediately: **a small model wastes a large GPU.** A 7B model on an 80 GB accelerator has the whole device to itself and uses a fraction of it. There are mechanisms to subdivide GPUs — time-slicing, MPS, and hardware partitioning such as MIG — but they are opt-in, come with real trade-offs, and none of them is the default. For LLM serving this is usually acceptable, because the standard approach is the opposite: rather than several models sharing a GPU, one model instance takes whole GPUs and serves many users through batching. Asking for “four GPUs” is rarely specific enough. A cluster may contain several accelerator generations, machines with different GPU counts, and — most importantly — different interconnects between the GPUs inside them. That last point matters more than it appears. A model sharded across four GPUs has those GPUs in constant communication during inference. Four GPUs linked by a fast in-server interconnect and four GPUs that merely happen to be on the same node are very different propositions for performance. Kubernetes gives you a few controls for this: **Labels and node selectors** — nodes are labelled with what they are (accelerator type, region, pool), and Pods select the labels they need. **Affinity rules** — a richer form of the same idea, including *anti*-affinity, which is how you keep the replicas of one Deployment from all landing on the same node and dying together with it. **Taints and tolerations** — an inversion of the above. A taint marks a node as repellent by default; only Pods that explicitly tolerate it are placed there. GPU nodes are almost always tainted, because otherwise ordinary CPU workloads would drift onto your expensive hardware and occupy it. Taints keep the wrong workloads off GPU nodes. Selectors and affinity put the right ones on the right GPU nodes. A question that comes up as soon as models get large: if a model is sharded across eight GPUs, is that eight Pods or one? Usually one. A Pod requests all the GPUs its model instance needs, and the inference engine coordinates them internally as a single logical server. Eight GPUs in one Pod, one address, one endpoint. That works because a **Pod cannot span machines**. It is scheduled onto exactly one node, so this arrangement holds only while the whole model fits in a single server’s GPUs. When a model needs more GPUs than any one machine has, that breaks. The replica must now span nodes, which means multiple Pods — a leader and its workers — that together form **one** model instance. And that is a genuinely awkward thing to express, because a Deployment is built for identical, independent, interchangeable Pods, and these are none of those. They have distinct roles, they must all be running before any of them is useful, and if one dies the whole group is broken rather than four-fifths working. Kubernetes has grown purpose-built resources for exactly this shape of workload, which schedule a leader-plus-workers group as a single unit and restart it as a unit. The takeaway for now is the constraint itself: A single-node model replica is one Pod. A multi-node replica is a group of Pods that must be created, scheduled and replaced together. This is also why the topology concerns from the previous section matter so much. Keeping a replica inside one node keeps it inside one Pod, and keeps the whole arrangement far simpler. A near-universal first experience, and worth understanding because the cause is structural. Pending means the scheduler cannot find a node satisfying the Pod's requirements. For CPU workloads this is usually a mild shortage that resolves as things shift around. For GPU workloads it is often absolute, and indivisibility is why. A Pod requesting eight GPUs needs **eight free GPUs on a single node**. A cluster with sixteen idle GPUs spread two-per-node across eight machines cannot schedule it. There is plenty of capacity in aggregate and none in the shape required. The same fragmentation problem that appears inside GPU memory reappears at cluster scale. Free capacity is not useful unless it is the right shape. Kubernetes checks Pods with **probes**, and the distinction between two of them is the most operationally important thing in this article. **Liveness** asks: is this process still functioning? If it fails, the Pod is killed and replaced. **Readiness** asks: should this Pod receive traffic right now? If it fails, the Pod stays alive but is removed from the pool of destinations. This is the mechanism the router from the previous layer ultimately depends on. Now the LLM-specific part. A web application container starts in seconds. An inference Pod does not: This takes **minutes**, not seconds. Where the weights come from is worth a note, because it is the stage that dominates. Baking them into the container image makes the image enormous — pulling hundreds of gigabytes onto a fresh node is slow, and every new replica pays for it. The common alternative is to keep the image lean and fetch weights at startup from object storage or a shared volume, with a node-local cache so that later Pods on the same machine start much faster than the first. Either way, this step is usually the reason a replica takes minutes rather than seconds to appear, and it is the first place to look when scale-up feels too slow. The startup time creates two requirements. First, readiness must gate traffic strictly. A Pod that is running but still loading weights will fail every request sent to it. Readiness is what prevents that, and it is why **“the Pod is up”** and **“the Pod can serve”** are entirely different statements for inference workloads. Second, liveness must not fire during startup. A liveness probe with a short timeout will decide the still-loading Pod is broken, kill it, and start a fresh one — which will also be killed, forever. This is why **startup probes** exist: they suspend liveness checking until the process has finished coming up. Getting this wrong produces an infinite crash loop that looks like a broken image and is actually a misconfigured timeout. Pods are disposable and their addresses change, so clients never target them directly. A **Service** provides a stable name and address in front of a changing set of Pods, and the control plane keeps its list of endpoints updated as Pods become ready or stop being ready. This is exactly the list the router needed. Readiness feeds endpoints; endpoints feed whatever routes traffic. Which raises the obvious question: **a Service already load-balances across Pods, so why is a separate router needed at all?** Because a Service balances by connection, in effect round robin. That is the precise behaviour that fails for inference — it cannot tell a 200-token request from a 30,000-token one, and knows nothing about which replica holds useful cached state. So a production deployment usually has both, doing different jobs: The router is a normal workload in the cluster. It uses Kubernetes to *discover* which server Pods are ready, then makes its own decision about which one to use — deliberately bypassing the Service’s built-in balancing. Kubernetes supplies membership; the router supplies judgement. Deploying a change means replacing Pods, and Kubernetes does this gradually — bring up new Pods, wait for readiness, retire old ones. For a web service it is routine. For inference, three things bite. **Surge costs GPUs.** A rolling update starts new Pods before removing old ones, so during a rollout you need capacity for both. When a Pod holds four GPUs, surging one replica means four additional GPUs must be free. On a full cluster the rollout simply stalls, with new Pods Pending. **Rollouts are slow.** Each new Pod takes minutes to become ready, and they proceed in batches. A rollout across many replicas is measured in tens of minutes. **In-flight requests need draining.** When a Pod is told to stop, Kubernetes gives it a grace period and then kills it. The default is thirty seconds, which is fine for HTTP requests that finish in milliseconds — and not fine for a generation that has been streaming for a minute and is not done. Too short a grace period cuts responses off mid-sentence. Long-running generation needs a longer window and a server that stops accepting new work while finishing what it has. A Pod crashes — out-of-memory, a hardware fault, a node drained for maintenance. Kubernetes notices the desired state is unmet and creates a replacement. That replacement is correct in every respect Kubernetes cares about. It is the same image, the same configuration, the same GPU count. It is also completely cold. Everything the dead Pod had accumulated in GPU memory is gone: The replacement Pod starts with an empty cache and takes minutes to reach ready. Once it does, it begins recomputing prefixes that were, until recently, already sitting in memory a few racks away. Every request routed to it during that window pays full prefill cost. A replaced Pod is functionally identical and economically much worse, for as long as it takes to warm up*.* Two practical implications. **Restarts are expensive in a way Kubernetes does not model** — it sees a healthy replacement, while your latency percentiles see a cold one. And because Pods are treated as interchangeable, nothing in the orchestration layer knows that the Pod it just terminated was the one holding your hottest prefix. Kubernetes can add replicas as load rises. The standard mechanism scales on CPU utilisation, which for inference is close to meaningless — the CPU is doing tokenisation and request handling while the GPU does the work. A saturated inference server can show modest CPU usage. So scaling must be driven by signals from the inference server itself. The useful ones are the same ones a router uses: **queue depth in tokens**, **KV cache utilisation**, **running sequences**, and **TTFT against your target**. Cache utilisation is often the best single trigger, because it is the resource that actually runs out. Then there is the timing problem, which is more fundamental. Scaling out a web service takes seconds, so you can react to load as it arrives. Adding an inference replica takes minutes — and if no GPU node is free, the cluster autoscaler must provision a new machine first, which takes longer still. By the time capacity arrives, the traffic spike may be over. The consequences are unavoidable rather than clever. You **over-provision**, keeping headroom that costs money in order to absorb spikes. You **scale on leading indicators** — queue growth rather than saturation — to buy back some of the delay. You **scale down conservatively**, because removing a replica discards a warm cache you will pay minutes to rebuild. And where traffic is predictable, you scale on a schedule rather than reacting at all. Kubernetes gives an inference platform a great deal. Pods land on machines with the right accelerators and the right topology. Dead replicas are replaced without anyone waking up. Readiness keeps traffic away from servers that cannot serve. Rollouts happen gradually and can be reversed. Capacity follows load, if slowly. Most valuable of all, there is one authoritative, continuously updated answer to *which inference servers exist and which are ready* — which is exactly what the routing layer required. What it does not give you is any understanding of the workload. Kubernetes schedules on the resources you declared when you deployed: GPUs, CPU, memory. Those numbers are fixed at deploy time and describe the container, not what is happening inside it. It has no concept of a token, a batch, a KV cache block or a cached prefix. A Pod at 95% cache utilisation and a Pod at 10% are, to the control plane, two healthy Pods. That is not a criticism. It is a design decision that makes Kubernetes general enough to run anything. But it means the orchestration layer and the routing layer each hold half of the truth, and neither holds both. Look at what we have actually built. Kubernetes knows which Pods exist, which are ready, and where they run. It knows nothing about what is inside them. The router knows what is inside them — cache utilisation, queue depth, which prefixes are warm. It knows nothing about creating, replacing or scaling them. So the two most important decisions in the system are made by components that cannot see each other’s information. Kubernetes terminates a Pod during a rollout without knowing it held the hottest prefix cache in the fleet. It scales up on a metric it does not understand, delivered by a custom pipeline someone had to build. Its Service load-balances in exactly the way we established is wrong, so we bypass it with a router that lives beside the platform rather than within it. Every one of these is a workaround. Each one works, and each one exists because the orchestration layer models an inference server as a generic container with a GPU attached. The alternative is to make orchestration itself inference-aware — for the platform to understand that these Pods serve models, hold caches, and have a load that is measured in tokens rather than connections. That means treating a pool of inference servers as a first-class object, routing with knowledge of what each holds, scheduling with knowledge of what each is doing, and being able to place the two phases of a request on different hardware when that helps. **What does an orchestration layer look like when it is built for LLM inference specifically?** That is the last architectural piece of the stack. **1. What really happens when you click ‘Send’ on ChatGPT** — A journey through modern AI Infrastructure **2. What Do You Do With a Model That’s Too Big for Your GPU?** — Quantization, Sharding and Parallelism Explained **3. How Does One GPU Serve Hundreds of Users at the Same Time?** — Inside an LLM inference server **4. The KV Cache Explained: Why Long Conversations Get Expensive** — How LLMs remember context without recomputing everything **5. Why Is Your LLM Recomputing the Same Prompt 1,000 Times a Day?** — Prefix caching, radix trees and block hashing explained **6. Why Traditional Load Balancing Breaks for LLMs** — Building an LLM-aware router **7. Kubernetes for LLM Inference: How AI Workloads Run Across a GPU Cluster** **8. LLM-D Explained** — How modern AI infrastructure routes, schedules and scales LLM inference (Next Article) **9. Inside a Modern AI Inference Platform** — The full stack end-to-end **Sources** [Kubernetes for LLM Inference: How AI Workloads Run Across a GPU Cluster](https://pub.towardsai.net/kubernetes-for-llm-inference-how-ai-workloads-run-across-a-gpu-cluster-424f0dcfab8c) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.