{"slug": "running-ai-workloads-on-kubernetes-gpus-scheduling-scaling-and-model-serving", "title": "Running AI Workloads on Kubernetes: GPUs, Scheduling, Scaling, and Model Serving", "summary": "A developer explains how Kubernetes scheduling, autoscaling, and model-serving patterns must adapt to handle GPU-accelerated AI workloads. The post highlights challenges such as gang scheduling, topology-aware placement, and cost-efficient GPU utilization, noting that Kubernetes v1.36 introduces features like PodGroups and Dynamic Resource Allocation to address these needs.", "body_md": "In Part 1 of **AI Infrastructure for Cloud Engineers**, we looked at why Kubernetes is becoming an important foundation for production AI systems.\n\nRead Part 1:\n\n[Why Kubernetes Is Becoming the Operating System for AI Infrastructure]\n\nNow the question is: **what actually changes when we start running AI workloads on Kubernetes?**\n\nTraditional applications are usually scheduled around familiar resources such as CPU, memory, storage, and network capacity.\n\nAI workloads introduce another resource that changes the infrastructure equation:\n\n```\nGPU\n```\n\nGPUs are powerful, expensive, and limited. Once teams start running model inference, embedding services, fine-tuning jobs, or other AI workloads at scale, simply deploying a container is no longer enough.\n\nThe platform also needs to decide:\n\nThis is where Kubernetes scheduling, autoscaling, and model-serving patterns start to matter.\n\nLet's look at how these pieces fit together.\n\nKubernetes normally schedules Pods based on resources such as CPU and memory.\n\nA basic application might request:\n\n```\nresources:\n  requests:\n    cpu: \"500m\"\n    memory: \"1Gi\"\n  limits:\n    cpu: \"1\"\n    memory: \"2Gi\"\n```\n\nGPU-enabled nodes add specialized resources to the cluster.\n\nA workload can then request a GPU:\n\n```\nresources:\n  limits:\n    nvidia.com/gpu: 1\n```\n\nConceptually, the cluster might look like this:\n\n```\nKubernetes Cluster\n\nNode A\n├── CPU\n├── Memory\n└── No GPU\n\nNode B\n├── CPU\n├── Memory\n└── GPU\n\nNode C\n├── CPU\n├── Memory\n└── GPU\n```\n\nIf an AI workload requests a GPU, Kubernetes needs to place it on a node where that resource is available.\n\nHardware vendors commonly expose devices such as GPUs to Kubernetes through mechanisms including device plugins.\n\nThat sounds straightforward.\n\nAt larger scale, however, GPU scheduling becomes much more interesting.\n\nImagine a cluster with multiple accelerator types.\n\n```\nNode A → NVIDIA T4\nNode B → NVIDIA A100\nNode C → NVIDIA H100\nNode D → CPU only\n```\n\nNow imagine three workloads:\n\n```\nSmall embedding model\nLarge language model\nDistributed training job\n```\n\nPlacing all three randomly would be inefficient.\n\nThe embedding workload may not need the most powerful GPU, while the large model may require significantly more accelerator memory and compute.\n\nThis means AI platforms often need to consider:\n\n```\nGPU type\nGPU memory\nWorkload size\nTopology\nAvailability\nCost\nPriority\n```\n\nModern Kubernetes scheduling is evolving specifically for these kinds of workloads.\n\nKubernetes v1.36, for example, introduced further workload-aware scheduling capabilities including PodGroups, topology-aware scheduling, workload-aware preemption, and integration with Dynamic Resource Allocation. These features are particularly relevant to tightly coupled AI/ML and batch workloads.\n\nTraditional Kubernetes scheduling largely thinks about individual Pods.\n\nAI workloads may need Kubernetes to think about a **group of Pods together**.\n\nImagine distributed training that requires four workers:\n\n```\nTraining Job\n\nWorker 1\nWorker 2\nWorker 3\nWorker 4\n```\n\nScheduling only two workers while the others remain pending may not be useful if the job requires all four before it can start.\n\nThis is the idea behind **gang scheduling**.\n\n```\nEnough resources for all workers?\n\n        Yes\n         ↓\nSchedule workload\n\n        No\n         ↓\nWait for capacity\n```\n\nTopology can also matter.\n\nIf several workers constantly exchange large amounts of data, placing them far apart across the infrastructure may introduce unnecessary network overhead.\n\nWorkload-aware and topology-aware scheduling allow Kubernetes to make placement decisions using more context about the complete workload rather than treating every Pod independently.\n\nGPUs can represent a significant portion of the infrastructure cost behind self-hosted AI.\n\nThat makes low utilization expensive.\n\nImagine:\n\n```\nGPU Capacity\n████████████████████ 100%\n\nActual Workload\n██████               30%\n```\n\nThe remaining capacity is still being paid for.\n\nThis can happen when:\n\nOne goal of an AI platform is therefore not just:\n\nMake the model run.\n\nIt is:\n\nKeep the model responsive while using expensive compute efficiently.\n\nThe cloud-native ecosystem is increasingly developing GPU-sharing and accelerator-aware scheduling approaches for this reason. For example, HAMi focuses on sharing and scheduling heterogeneous accelerator resources, while Kubernetes Dynamic Resource Allocation provides a more flexible mechanism for requesting specialized devices.\n\nAI infrastructure discussions often combine training and inference, but they have different operational characteristics.\n\nTraining commonly looks like:\n\n```\nDataset\n   ↓\nTraining Job\n   ↓\nMany GPUs\n   ↓\nHours / Days\n   ↓\nModel\n```\n\nThe workload may require several accelerators simultaneously and run for a long period.\n\nInference looks more like:\n\n```\nUser Request\n      ↓\nModel Server\n      ↓\nGPU\n      ↓\nGenerated Response\n```\n\nInference is usually much more sensitive to:\n\n```\nLatency\nThroughput\nAvailability\nQueue depth\nConcurrent requests\n```\n\nFor a user-facing AI application, a model that eventually returns the correct answer is not enough.\n\nIt also needs to respond within an acceptable amount of time.\n\nThat changes how we think about scaling.\n\nFor many web applications, Kubernetes autoscaling might use CPU utilization.\n\n```\nCPU > 70%\n     ↓\nAdd Pods\n```\n\nThat can work well for traditional services.\n\nAI inference may need different signals.\n\nImagine an inference server where:\n\n```\nCPU = 35%\nGPU = 92%\nWaiting requests = 120\n```\n\nFrom CPU alone, the application may appear healthy.\n\nFrom the user's perspective, it may already be overloaded.\n\nBetter AI scaling signals may include:\n\n```\nGPU utilization\nRequests waiting\nConcurrent requests\nInference latency\nTokens per second\nKV cache utilization\nQueue depth\n```\n\nKServe, for example, supports autoscaling inference workloads using external LLM metrics through technologies such as KEDA, Prometheus, and OpenTelemetry. Its documentation includes examples based on active or waiting inference requests rather than relying only on CPU.\n\nA simplified scaling flow could look like this:\n\n```\nRequest Queue\n     ↓\nWaiting requests increase\n     ↓\nAutoscaling signal\n     ↓\nCreate more inference replicas\n     ↓\nMore capacity available\n```\n\nA trained model is essentially an artifact.\n\nUsers still need a service capable of loading the model and accepting requests.\n\nThat layer is commonly called **model serving**.\n\nConceptually:\n\n```\nApplication\n     ↓\nModel Endpoint\n     ↓\nInference Server\n     ↓\nModel\n     ↓\nGPU\n```\n\nA production model-serving layer may need to handle:\n\nInstead of application developers building all of this independently, model-serving frameworks can provide reusable infrastructure.\n\nOne Kubernetes-native example is **KServe**, which provides abstractions for deploying and operating inference workloads on Kubernetes.\n\nThe wider cloud-native ecosystem is also building more specialized inference infrastructure. Kubernetes' former WG Serving helped advance inference-oriented capabilities including request scheduling and gateway patterns before concluding its work in 2026.\n\nPutting the pieces together, an inference platform might look like this:\n\n```\n                    Users\n                      ↓\n               API / AI Gateway\n                      ↓\n                Request Router\n                      ↓\n          ┌───────────┼───────────┐\n          ↓           ↓           ↓\n     Model Pod    Model Pod    Model Pod\n          ↓           ↓           ↓\n        GPU         GPU         GPU\n\n              Kubernetes Cluster\n                      ↓\n        ┌─────────────┼─────────────┐\n        ↓             ↓             ↓\n   Autoscaling    Monitoring     Scheduling\n```\n\nKubernetes handles the infrastructure layer.\n\nThe model-serving layer handles inference-specific concerns.\n\nTogether, they allow the platform to respond to changing demand.\n\nBasic load balancing assumes that multiple application replicas are roughly interchangeable.\n\nAI inference can be different.\n\nThe best place to route a request may depend on:\n\nSo instead of:\n\n```\nRequest\n   ↓\nRandom Pod\n```\n\nAI-aware routing can move toward:\n\n```\nRequest\n   ↓\nInference Gateway\n   ↓\nBest available model server\n```\n\nModern cloud-native inference projects are increasingly exploring model-aware and state-aware routing.\n\nFor example, llm-d focuses on capabilities such as inference scheduling, KV-cache-aware behavior, and separating prompt processing from token generation to improve resource utilization and inference performance.\n\nThis is one of the clearest examples of Kubernetes infrastructure adapting specifically to AI workloads.\n\nThere is an important limitation to remember.\n\nSuppose Kubernetes scales an inference application:\n\n```\n2 replicas\n    ↓\n4 replicas\n    ↓\n8 replicas\n```\n\nThose eight replicas may now generate much more traffic toward:\n\n```\nVector database\nObject storage\nExternal APIs\nModel storage\nNetwork\nGPU nodes\n```\n\nScaling one component can simply move the bottleneck somewhere else.\n\nFor example:\n\n```\nInference Pods\n████████████████  Healthy\n\n        ↓\n\nVector Database\n████████████████  Overloaded\n```\n\nCapacity planning needs to consider the complete request path.\n\nThis is the same lesson cloud engineers already know from distributed systems.\n\nAI does not remove bottlenecks.\n\nIt introduces some new ones.\n\nSuppose a GPU node fails while running an inference workload.\n\nThe platform needs to detect that condition and recover.\n\nA production architecture should consider:\n\n```\nPod failures\nNode failures\nGPU failures\nModel loading failures\nProvider failures\nNetwork failures\nOut-of-memory conditions\n```\n\nKubernetes can restart or reschedule workloads, but AI platforms also need visibility into accelerator health and inference behavior.\n\nRecent Kubernetes Dynamic Resource Allocation work includes exposing device health information to workloads and controllers, which can help operators understand failures involving specialized hardware.\n\nThe key point is simple:\n\n```\nProcess running\n        ≠\nAI service healthy\n```\n\nInfrastructure health and application health both matter.\n\nFor an AI workload running on Kubernetes, I would separate metrics into three layers.\n\n```\nPod availability\nPod restarts\nNode health\nCPU\nMemory\nNetwork\nGPU utilization\nGPU memory\nAccelerator availability\nDevice health\nRequest latency\nQueue depth\nRequests running\nTokens per second\nTime to first token\nInference errors\n```\n\nLooking at only one layer can hide the real problem.\n\nFor example:\n\n```\nKubernetes\nPods healthy ✓\n\nGPU\nUtilization 100%\n\nInference\nLatency increasing ↑\nQueue growing ↑\n```\n\nThe cluster is technically running.\n\nThe service is still degrading.\n\nYou do not need to become a machine-learning researcher to work with AI infrastructure.\n\nThe infrastructure problems remain very familiar:\n\n```\nScheduling\nScaling\nNetworking\nCapacity\nObservability\nSecurity\nReliability\nCost\n```\n\nThe difference is the resource being managed.\n\nInstead of only asking:\n\n```\nHow much CPU?\nHow much memory?\n```\n\nwe now also ask:\n\n```\nWhich GPU?\nHow much GPU memory?\nWhich model?\nHow many concurrent requests?\nHow many tokens per second?\nWhere should this inference request run?\n```\n\nThat is the bridge between traditional cloud engineering and AI infrastructure.\n\nBefore running AI workloads on Kubernetes, think about:\n\nGPUs, scheduling, and model serving solve only part of the production problem.\n\nOnce the application is running, the next question becomes:\n\nHow do we know whether the AI system is actually healthy?\n\nTraditional infrastructure monitoring gives us CPU, memory, and Pod health.\n\nAI workloads introduce another set of signals including inference latency, token throughput, GPU utilization, queue depth, model failures, and cost.\n\nThat is what we will cover next.\n\nPart 3: Observability for AI Infrastructure: What to Monitor Beyond CPU and Memory\n\nRunning AI workloads on Kubernetes is not simply a matter of adding a GPU to a Pod.\n\nProduction systems need to think about the complete lifecycle:\n\n```\nGPU Allocation\n      ↓\nScheduling\n      ↓\nModel Serving\n      ↓\nRequest Routing\n      ↓\nAutoscaling\n      ↓\nObservability\n      ↓\nFailure Recovery\n```\n\nKubernetes gives us a strong orchestration foundation.\n\nBut AI introduces new constraints around expensive accelerators, workload placement, inference latency, and resource utilization.\n\nThe interesting shift is that Kubernetes is beginning to understand more about these workloads directly, while projects around it are adding the inference-specific capabilities required to operate AI efficiently.\n\nFor cloud engineers, this is where existing Kubernetes knowledge starts becoming directly useful in the AI infrastructure world.\n\nThis article is **Part 2 of my AI Infrastructure for Cloud Engineers series**:\n\nI regularly share what I learn about cloud infrastructure, Kubernetes, DevOps, SRE, and the engineering behind production AI systems.\n\nLooking forward to connect, learn and grow together 😄\n\n**LinkedIn:** [Connect with me on LinkedIn](https://www.linkedin.com/in/sushyamnagallapati/)\n\nIf you're running AI workloads on Kubernetes, what has been harder in practice: GPU allocation, autoscaling, model serving, or keeping the GPUs efficiently utilized?", "url": "https://wpnews.pro/news/running-ai-workloads-on-kubernetes-gpus-scheduling-scaling-and-model-serving", "canonical_source": "https://dev.to/sushyam_nagallapati/running-ai-workloads-on-kubernetes-gpus-scheduling-scaling-and-model-serving-5edh", "published_at": "2026-08-17 09:15:00+00:00", "updated_at": "2026-08-17 09:42:42.035772+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-infrastructure", "mlops", "developer-tools"], "entities": ["Kubernetes", "NVIDIA", "PodGroups", "Dynamic Resource Allocation"], "alternates": {"html": "https://wpnews.pro/news/running-ai-workloads-on-kubernetes-gpus-scheduling-scaling-and-model-serving", "markdown": "https://wpnews.pro/news/running-ai-workloads-on-kubernetes-gpus-scheduling-scaling-and-model-serving.md", "text": "https://wpnews.pro/news/running-ai-workloads-on-kubernetes-gpus-scheduling-scaling-and-model-serving.txt", "jsonld": "https://wpnews.pro/news/running-ai-workloads-on-kubernetes-gpus-scheduling-scaling-and-model-serving.jsonld"}}