Running AI Workloads on Kubernetes: GPUs, Scheduling, Scaling, and Model Serving

A developer explains how Kubernetes scheduling, autoscaling, and model-serving patterns must adapt to handle GPU-accelerated AI workloads. The post highlights challenges such as gang scheduling, topology-aware placement, and cost-efficient GPU utilization, noting that Kubernetes v1.36 introduces features like PodGroups and Dynamic Resource Allocation to address these needs.

In Part 1 of AI Infrastructure for Cloud Engineers , we looked at why Kubernetes is becoming an important foundation for production AI systems. Read Part 1: Why Kubernetes Is Becoming the Operating System for AI Infrastructure Now the question is: what actually changes when we start running AI workloads on Kubernetes? Traditional applications are usually scheduled around familiar resources such as CPU, memory, storage, and network capacity. AI workloads introduce another resource that changes the infrastructure equation: GPU GPUs are powerful, expensive, and limited. Once teams start running model inference, embedding services, fine-tuning jobs, or other AI workloads at scale, simply deploying a container is no longer enough. The platform also needs to decide: This is where Kubernetes scheduling, autoscaling, and model-serving patterns start to matter. Let's look at how these pieces fit together. Kubernetes normally schedules Pods based on resources such as CPU and memory. A basic application might request: resources: requests: cpu: "500m" memory: "1Gi" limits: cpu: "1" memory: "2Gi" GPU-enabled nodes add specialized resources to the cluster. A workload can then request a GPU: resources: limits: nvidia.com/gpu: 1 Conceptually, the cluster might look like this: Kubernetes Cluster Node A ├── CPU ├── Memory └── No GPU Node B ├── CPU ├── Memory └── GPU Node C ├── CPU ├── Memory └── GPU If an AI workload requests a GPU, Kubernetes needs to place it on a node where that resource is available. Hardware vendors commonly expose devices such as GPUs to Kubernetes through mechanisms including device plugins. That sounds straightforward. At larger scale, however, GPU scheduling becomes much more interesting. Imagine a cluster with multiple accelerator types. Node A → NVIDIA T4 Node B → NVIDIA A100 Node C → NVIDIA H100 Node D → CPU only Now imagine three workloads: Small embedding model Large language model Distributed training job Placing all three randomly would be inefficient. The embedding workload may not need the most powerful GPU, while the large model may require significantly more accelerator memory and compute. This means AI platforms often need to consider: GPU type GPU memory Workload size Topology Availability Cost Priority Modern Kubernetes scheduling is evolving specifically for these kinds of workloads. Kubernetes v1.36, for example, introduced further workload-aware scheduling capabilities including PodGroups, topology-aware scheduling, workload-aware preemption, and integration with Dynamic Resource Allocation. These features are particularly relevant to tightly coupled AI/ML and batch workloads. Traditional Kubernetes scheduling largely thinks about individual Pods. AI workloads may need Kubernetes to think about a group of Pods together . Imagine distributed training that requires four workers: Training Job Worker 1 Worker 2 Worker 3 Worker 4 Scheduling only two workers while the others remain pending may not be useful if the job requires all four before it can start. This is the idea behind gang scheduling . Enough resources for all workers? Yes ↓ Schedule workload No ↓ Wait for capacity Topology can also matter. If several workers constantly exchange large amounts of data, placing them far apart across the infrastructure may introduce unnecessary network overhead. Workload-aware and topology-aware scheduling allow Kubernetes to make placement decisions using more context about the complete workload rather than treating every Pod independently. GPUs can represent a significant portion of the infrastructure cost behind self-hosted AI. That makes low utilization expensive. Imagine: GPU Capacity ████████████████████ 100% Actual Workload ██████ 30% The remaining capacity is still being paid for. This can happen when: One goal of an AI platform is therefore not just: Make the model run. It is: Keep the model responsive while using expensive compute efficiently. The cloud-native ecosystem is increasingly developing GPU-sharing and accelerator-aware scheduling approaches for this reason. For example, HAMi focuses on sharing and scheduling heterogeneous accelerator resources, while Kubernetes Dynamic Resource Allocation provides a more flexible mechanism for requesting specialized devices. AI infrastructure discussions often combine training and inference, but they have different operational characteristics. Training commonly looks like: Dataset ↓ Training Job ↓ Many GPUs ↓ Hours / Days ↓ Model The workload may require several accelerators simultaneously and run for a long period. Inference looks more like: User Request ↓ Model Server ↓ GPU ↓ Generated Response Inference is usually much more sensitive to: Latency Throughput Availability Queue depth Concurrent requests For a user-facing AI application, a model that eventually returns the correct answer is not enough. It also needs to respond within an acceptable amount of time. That changes how we think about scaling. For many web applications, Kubernetes autoscaling might use CPU utilization. CPU 70% ↓ Add Pods That can work well for traditional services. AI inference may need different signals. Imagine an inference server where: CPU = 35% GPU = 92% Waiting requests = 120 From CPU alone, the application may appear healthy. From the user's perspective, it may already be overloaded. Better AI scaling signals may include: GPU utilization Requests waiting Concurrent requests Inference latency Tokens per second KV cache utilization Queue depth KServe, for example, supports autoscaling inference workloads using external LLM metrics through technologies such as KEDA, Prometheus, and OpenTelemetry. Its documentation includes examples based on active or waiting inference requests rather than relying only on CPU. A simplified scaling flow could look like this: Request Queue ↓ Waiting requests increase ↓ Autoscaling signal ↓ Create more inference replicas ↓ More capacity available A trained model is essentially an artifact. Users still need a service capable of loading the model and accepting requests. That layer is commonly called model serving . Conceptually: Application ↓ Model Endpoint ↓ Inference Server ↓ Model ↓ GPU A production model-serving layer may need to handle: Instead of application developers building all of this independently, model-serving frameworks can provide reusable infrastructure. One Kubernetes-native example is KServe , which provides abstractions for deploying and operating inference workloads on Kubernetes. The wider cloud-native ecosystem is also building more specialized inference infrastructure. Kubernetes' former WG Serving helped advance inference-oriented capabilities including request scheduling and gateway patterns before concluding its work in 2026. Putting the pieces together, an inference platform might look like this: Users ↓ API / AI Gateway ↓ Request Router ↓ ┌───────────┼───────────┐ ↓ ↓ ↓ Model Pod Model Pod Model Pod ↓ ↓ ↓ GPU GPU GPU Kubernetes Cluster ↓ ┌─────────────┼─────────────┐ ↓ ↓ ↓ Autoscaling Monitoring Scheduling Kubernetes handles the infrastructure layer. The model-serving layer handles inference-specific concerns. Together, they allow the platform to respond to changing demand. Basic load balancing assumes that multiple application replicas are roughly interchangeable. AI inference can be different. The best place to route a request may depend on: So instead of: Request ↓ Random Pod AI-aware routing can move toward: Request ↓ Inference Gateway ↓ Best available model server Modern cloud-native inference projects are increasingly exploring model-aware and state-aware routing. For example, llm-d focuses on capabilities such as inference scheduling, KV-cache-aware behavior, and separating prompt processing from token generation to improve resource utilization and inference performance. This is one of the clearest examples of Kubernetes infrastructure adapting specifically to AI workloads. There is an important limitation to remember. Suppose Kubernetes scales an inference application: 2 replicas ↓ 4 replicas ↓ 8 replicas Those eight replicas may now generate much more traffic toward: Vector database Object storage External APIs Model storage Network GPU nodes Scaling one component can simply move the bottleneck somewhere else. For example: Inference Pods ████████████████ Healthy ↓ Vector Database ████████████████ Overloaded Capacity planning needs to consider the complete request path. This is the same lesson cloud engineers already know from distributed systems. AI does not remove bottlenecks. It introduces some new ones. Suppose a GPU node fails while running an inference workload. The platform needs to detect that condition and recover. A production architecture should consider: Pod failures Node failures GPU failures Model loading failures Provider failures Network failures Out-of-memory conditions Kubernetes can restart or reschedule workloads, but AI platforms also need visibility into accelerator health and inference behavior. Recent Kubernetes Dynamic Resource Allocation work includes exposing device health information to workloads and controllers, which can help operators understand failures involving specialized hardware. The key point is simple: Process running ≠ AI service healthy Infrastructure health and application health both matter. For an AI workload running on Kubernetes, I would separate metrics into three layers. Pod availability Pod restarts Node health CPU Memory Network GPU utilization GPU memory Accelerator availability Device health Request latency Queue depth Requests running Tokens per second Time to first token Inference errors Looking at only one layer can hide the real problem. For example: Kubernetes Pods healthy ✓ GPU Utilization 100% Inference Latency increasing ↑ Queue growing ↑ The cluster is technically running. The service is still degrading. You do not need to become a machine-learning researcher to work with AI infrastructure. The infrastructure problems remain very familiar: Scheduling Scaling Networking Capacity Observability Security Reliability Cost The difference is the resource being managed. Instead of only asking: How much CPU? How much memory? we now also ask: Which GPU? How much GPU memory? Which model? How many concurrent requests? How many tokens per second? Where should this inference request run? That is the bridge between traditional cloud engineering and AI infrastructure. Before running AI workloads on Kubernetes, think about: GPUs, scheduling, and model serving solve only part of the production problem. Once the application is running, the next question becomes: How do we know whether the AI system is actually healthy? Traditional infrastructure monitoring gives us CPU, memory, and Pod health. AI workloads introduce another set of signals including inference latency, token throughput, GPU utilization, queue depth, model failures, and cost. That is what we will cover next. Part 3: Observability for AI Infrastructure: What to Monitor Beyond CPU and Memory Running AI workloads on Kubernetes is not simply a matter of adding a GPU to a Pod. Production systems need to think about the complete lifecycle: GPU Allocation ↓ Scheduling ↓ Model Serving ↓ Request Routing ↓ Autoscaling ↓ Observability ↓ Failure Recovery Kubernetes gives us a strong orchestration foundation. But AI introduces new constraints around expensive accelerators, workload placement, inference latency, and resource utilization. The interesting shift is that Kubernetes is beginning to understand more about these workloads directly, while projects around it are adding the inference-specific capabilities required to operate AI efficiently. For cloud engineers, this is where existing Kubernetes knowledge starts becoming directly useful in the AI infrastructure world. This article is Part 2 of my AI Infrastructure for Cloud Engineers series : I regularly share what I learn about cloud infrastructure, Kubernetes, DevOps, SRE, and the engineering behind production AI systems. Looking forward to connect, learn and grow together 😄 LinkedIn: Connect with me on LinkedIn https://www.linkedin.com/in/sushyamnagallapati/ If you're running AI workloads on Kubernetes, what has been harder in practice: GPU allocation, autoscaling, model serving, or keeping the GPUs efficiently utilized?