In Part 1 of AI Infrastructure for Cloud Engineers, we looked at why Kubernetes is becoming an important foundation for production AI systems.
Read Part 1:
[Why Kubernetes Is Becoming the Operating System for AI Infrastructure]
Now the question is: what actually changes when we start running AI workloads on Kubernetes?
Traditional applications are usually scheduled around familiar resources such as CPU, memory, storage, and network capacity.
AI workloads introduce another resource that changes the infrastructure equation:
GPU
GPUs are powerful, expensive, and limited. Once teams start running model inference, embedding services, fine-tuning jobs, or other AI workloads at scale, simply deploying a container is no longer enough.
The platform also needs to decide:
This is where Kubernetes scheduling, autoscaling, and model-serving patterns start to matter.
Let's look at how these pieces fit together.
Kubernetes normally schedules Pods based on resources such as CPU and memory.
A basic application might request:
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "1"
memory: "2Gi"
GPU-enabled nodes add specialized resources to the cluster.
A workload can then request a GPU:
resources:
limits:
nvidia.com/gpu: 1
Conceptually, the cluster might look like this:
Kubernetes Cluster
Node A
├── CPU
├── Memory
└── No GPU
Node B
├── CPU
├── Memory
└── GPU
Node C
├── CPU
├── Memory
└── GPU
If an AI workload requests a GPU, Kubernetes needs to place it on a node where that resource is available.
Hardware vendors commonly expose devices such as GPUs to Kubernetes through mechanisms including device plugins.
That sounds straightforward.
At larger scale, however, GPU scheduling becomes much more interesting.
Imagine a cluster with multiple accelerator types.
Node A → NVIDIA T4
Node B → NVIDIA A100
Node C → NVIDIA H100
Node D → CPU only
Now imagine three workloads:
Small embedding model
Large language model
Distributed training job
Placing all three randomly would be inefficient.
The embedding workload may not need the most powerful GPU, while the large model may require significantly more accelerator memory and compute.
This means AI platforms often need to consider:
GPU type
GPU memory
Workload size
Topology
Availability
Cost
Priority
Modern Kubernetes scheduling is evolving specifically for these kinds of workloads.
Kubernetes v1.36, for example, introduced further workload-aware scheduling capabilities including PodGroups, topology-aware scheduling, workload-aware preemption, and integration with Dynamic Resource Allocation. These features are particularly relevant to tightly coupled AI/ML and batch workloads.
Traditional Kubernetes scheduling largely thinks about individual Pods.
AI workloads may need Kubernetes to think about a group of Pods together.
Imagine distributed training that requires four workers:
Training Job
Worker 1
Worker 2
Worker 3
Worker 4
Scheduling only two workers while the others remain pending may not be useful if the job requires all four before it can start.
This is the idea behind gang scheduling.
Enough resources for all workers?
Yes
↓
Schedule workload
No
↓
Wait for capacity
Topology can also matter.
If several workers constantly exchange large amounts of data, placing them far apart across the infrastructure may introduce unnecessary network overhead.
Workload-aware and topology-aware scheduling allow Kubernetes to make placement decisions using more context about the complete workload rather than treating every Pod independently.
GPUs can represent a significant portion of the infrastructure cost behind self-hosted AI.
That makes low utilization expensive.
Imagine:
GPU Capacity
████████████████████ 100%
Actual Workload
██████ 30%
The remaining capacity is still being paid for.
This can happen when:
One goal of an AI platform is therefore not just:
Make the model run.
It is:
Keep the model responsive while using expensive compute efficiently.
The cloud-native ecosystem is increasingly developing GPU-sharing and accelerator-aware scheduling approaches for this reason. For example, HAMi focuses on sharing and scheduling heterogeneous accelerator resources, while Kubernetes Dynamic Resource Allocation provides a more flexible mechanism for requesting specialized devices.
AI infrastructure discussions often combine training and inference, but they have different operational characteristics.
Training commonly looks like:
Dataset
↓
Training Job
↓
Many GPUs
↓
Hours / Days
↓
Model
The workload may require several accelerators simultaneously and run for a long period.
Inference looks more like:
User Request
↓
Model Server
↓
GPU
↓
Generated Response
Inference is usually much more sensitive to:
Latency
Throughput
Availability
Queue depth
Concurrent requests
For a user-facing AI application, a model that eventually returns the correct answer is not enough.
It also needs to respond within an acceptable amount of time.
That changes how we think about scaling.
For many web applications, Kubernetes autoscaling might use CPU utilization.
CPU > 70%
↓
Add Pods
That can work well for traditional services.
AI inference may need different signals.
Imagine an inference server where:
CPU = 35%
GPU = 92%
Waiting requests = 120
From CPU alone, the application may appear healthy.
From the user's perspective, it may already be overloaded.
Better AI scaling signals may include:
GPU utilization
Requests waiting
Concurrent requests
Inference latency
Tokens per second
KV cache utilization
Queue depth
KServe, for example, supports autoscaling inference workloads using external LLM metrics through technologies such as KEDA, Prometheus, and OpenTelemetry. Its documentation includes examples based on active or waiting inference requests rather than relying only on CPU.
A simplified scaling flow could look like this:
Request Queue
↓
Waiting requests increase
↓
Autoscaling signal
↓
Create more inference replicas
↓
More capacity available
A trained model is essentially an artifact.
Users still need a service capable of the model and accepting requests.
That layer is commonly called model serving.
Conceptually:
Application
↓
Model Endpoint
↓
Inference Server
↓
Model
↓
GPU
A production model-serving layer may need to handle:
Instead of application developers building all of this independently, model-serving frameworks can provide reusable infrastructure.
One Kubernetes-native example is KServe, which provides abstractions for deploying and operating inference workloads on Kubernetes.
The wider cloud-native ecosystem is also building more specialized inference infrastructure. Kubernetes' former WG Serving helped advance inference-oriented capabilities including request scheduling and gateway patterns before concluding its work in 2026.
Putting the pieces together, an inference platform might look like this:
Users
↓
API / AI Gateway
↓
Request Router
↓
┌───────────┼───────────┐
↓ ↓ ↓
Model Pod Model Pod Model Pod
↓ ↓ ↓
GPU GPU GPU
Kubernetes Cluster
↓
┌─────────────┼─────────────┐
↓ ↓ ↓
Autoscaling Monitoring Scheduling
Kubernetes handles the infrastructure layer.
The model-serving layer handles inference-specific concerns.
Together, they allow the platform to respond to changing demand.
Basic load balancing assumes that multiple application replicas are roughly interchangeable.
AI inference can be different.
The best place to route a request may depend on:
So instead of:
Request
↓
Random Pod
AI-aware routing can move toward:
Request
↓
Inference Gateway
↓
Best available model server
Modern cloud-native inference projects are increasingly exploring model-aware and state-aware routing.
For example, llm-d focuses on capabilities such as inference scheduling, KV-cache-aware behavior, and separating prompt processing from token generation to improve resource utilization and inference performance.
This is one of the clearest examples of Kubernetes infrastructure adapting specifically to AI workloads.
There is an important limitation to remember.
Suppose Kubernetes scales an inference application:
2 replicas
↓
4 replicas
↓
8 replicas
Those eight replicas may now generate much more traffic toward:
Vector database
Object storage
External APIs
Model storage
Network
GPU nodes
Scaling one component can simply move the bottleneck somewhere else.
For example:
Inference Pods
████████████████ Healthy
↓
Vector Database
████████████████ Overloaded
Capacity planning needs to consider the complete request path.
This is the same lesson cloud engineers already know from distributed systems.
AI does not remove bottlenecks.
It introduces some new ones.
Suppose a GPU node fails while running an inference workload.
The platform needs to detect that condition and recover.
A production architecture should consider:
Pod failures
Node failures
GPU failures
Model failures
Provider failures
Network failures
Out-of-memory conditions
Kubernetes can restart or reschedule workloads, but AI platforms also need visibility into accelerator health and inference behavior.
Recent Kubernetes Dynamic Resource Allocation work includes exposing device health information to workloads and controllers, which can help operators understand failures involving specialized hardware.
The key point is simple:
Process running
≠
AI service healthy
Infrastructure health and application health both matter.
For an AI workload running on Kubernetes, I would separate metrics into three layers.
Pod availability
Pod restarts
Node health
CPU
Memory
Network
GPU utilization
GPU memory
Accelerator availability
Device health
Request latency
Queue depth
Requests running
Tokens per second
Time to first token
Inference errors
Looking at only one layer can hide the real problem.
For example:
Kubernetes
Pods healthy ✓
GPU
Utilization 100%
Inference
Latency increasing ↑
Queue growing ↑
The cluster is technically running.
The service is still degrading.
You do not need to become a machine-learning researcher to work with AI infrastructure.
The infrastructure problems remain very familiar:
Scheduling
Scaling
Networking
Capacity
Observability
Security
Reliability
Cost
The difference is the resource being managed.
Instead of only asking:
How much CPU?
How much memory?
we now also ask:
Which GPU?
How much GPU memory?
Which model?
How many concurrent requests?
How many tokens per second?
Where should this inference request run?
That is the bridge between traditional cloud engineering and AI infrastructure.
Before running AI workloads on Kubernetes, think about:
GPUs, scheduling, and model serving solve only part of the production problem.
Once the application is running, the next question becomes:
How do we know whether the AI system is actually healthy?
Traditional infrastructure monitoring gives us CPU, memory, and Pod health.
AI workloads introduce another set of signals including inference latency, token throughput, GPU utilization, queue depth, model failures, and cost.
That is what we will cover next.
Part 3: Observability for AI Infrastructure: What to Monitor Beyond CPU and Memory
Running AI workloads on Kubernetes is not simply a matter of adding a GPU to a Pod.
Production systems need to think about the complete lifecycle:
GPU Allocation
↓
Scheduling
↓
Model Serving
↓
Request Routing
↓
Autoscaling
↓
Observability
↓
Failure Recovery
Kubernetes gives us a strong orchestration foundation.
But AI introduces new constraints around expensive accelerators, workload placement, inference latency, and resource utilization.
The interesting shift is that Kubernetes is beginning to understand more about these workloads directly, while projects around it are adding the inference-specific capabilities required to operate AI efficiently.
For cloud engineers, this is where existing Kubernetes knowledge starts becoming directly useful in the AI infrastructure world.
This article is Part 2 of my AI Infrastructure for Cloud Engineers series:
I regularly share what I learn about cloud infrastructure, Kubernetes, DevOps, SRE, and the engineering behind production AI systems.
Looking forward to connect, learn and grow together 😄
LinkedIn: Connect with me on LinkedIn
If you're running AI workloads on Kubernetes, what has been harder in practice: GPU allocation, autoscaling, model serving, or keeping the GPUs efficiently utilized?