GPU Scheduling and Bin-Packing in Kubernetes: Pack More AI onto Every GPU

GPU utilization in Kubernetes clusters averages just 5% due to default scheduling that spreads pods across nodes instead of bin-packing them onto fewer GPUs. NVIDIA's device plugin and Dynamic Resource Allocation (DRA) in Kubernetes 1.34 enable denser packing, reducing costs from idle GPUs that can reach $4,954 per month per H100.

GPU scheduling decides which GPU a pod lands on. Bin-packing decides how densely GPUs fill up before new ones are added. Default Kubernetes handles neither well, which is a primary reason GPU utilization averages just 5% across production clusters. GPU-aware bin-packing, the NVIDIA device plugin, and Dynamic Resource Allocation place more AI work on fewer GPUs. This guide explains how each layer works and how to apply them. Key Takeaways - According to the Cast AI 2026 State of Kubernetes Optimization Report https://cast.ai/reports/state-of-kubernetes-optimization/ , average GPU utilization across tens of thousands of production clusters is 5%. CPU averages 8%, memory 20%. - The default scheduler uses LeastAllocated scoring, which spreads pods across many nodes instead of filling existing ones first. - The NVIDIA device plugin exposes GPUs as integer resources nvidia.com/gpu . It has no attribute awareness and no fractional allocation support. - MIG partitions one H100 into up to 7 hardware-isolated instances. Time-slicing creates up to 48 virtual replicas through CUDA context switching. - GPU-aware bin-packing requires three technical layers: right-sized requests, MostAllocated scheduler scoring, and ongoing consolidation. - Dynamic Resource Allocation DRA , stable in Kubernetes 1.34, replaces integer counting with attribute-based CEL filtering and supports device sharing across pods. How Kubernetes Schedules GPUs The Device Plugin Framework Kubernetes does not manage GPUs natively. Instead, it relies on the device plugin framework, which reached stable status in Kubernetes 1.26. Hardware vendors, including NVIDIA, AMD, and Intel, deploy DaemonSet-based plugins that register with the kubelet on each node. Each plugin advertises its devices as extended resources. For NVIDIA GPUs, that resource is nvidia.com/gpu . A pod requests a GPU with a single field in its spec: apiVersion: v1 kind: Pod metadata: name: gpu-job spec: containers: - name: trainer image: nvcr.io/nvidia/pytorch:24.01-py3 resources: limits: nvidia.com/gpu: "1" The scheduler sees an integer counter. It finds a node with at least one available nvidia.com/gpu unit and places the pod there. That is the complete default GPU scheduling logic. The NVIDIA GPU Operator The NVIDIA GPU Operator automates the full driver and plugin stack. It installs GPU drivers, the device plugin, the DCGM Exporter for metrics, the MIG Manager DaemonSet for partition management, and GPU Feature Discovery for node labeling. You deploy it via a single Helm chart. Four DCGM metrics matter most for utilization monitoring: DCGM FI DEV GPU UTIL : compute utilization percentage DCGM FI DEV FB USED / DCGM FI DEV FB FREE : framebuffer memory usage and headroom DCGM FI DEV SM CLOCK : streaming multiprocessor clock, which drops under thermal throttling DCGM FI DEV POWER USAGE : power draw These metrics show what the GPU actually does, not just what Kubernetes believes it is doing. That distinction matters for consolidation decisions. Why Default Scheduling Wastes GPUs LeastAllocated: The Spreading Problem The default scheduler scores nodes using LeastAllocated. It assigns higher scores to nodes with more free capacity. As a result, new pods consistently land on the emptiest available node. The effect is predictable: workloads spread thin across many nodes, and no single node fills up. For CPU workloads, this is a reasonable trade-off. For GPUs, it is expensive. An H100 on AWS costs roughly $6.50 to $7.00 per GPU per hour on demand. GCP pricing runs $9.00 to $11.50. Azure runs $11.00 to $13.00. Every half-empty node burns that rate with no return. An idle H100 costs approximately $4,954 per month. The Binary Allocation Problem The scheduler allocates based on requests, not actual usage. A pod requesting one GPU gets exclusive ownership of the entire physical device, regardless of whether it uses 10% or 90% of the hardware. There is no native fractional allocation. There is no attribute awareness. The scheduler cannot distinguish an A100 from an H100. It only counts. The Cast AI 2026 State of Kubernetes Optimization Report https://cast.ai/reports/state-of-kubernetes-optimization/ measured this problem across tens of thousands of production clusters: average GPU utilization is 5%. Laurent Gil, co-founder and president of Cast AI, summarizes it directly: “A GPU sitting idle costs dollars per hour. A CPU sitting idle costs cents. And 95% of GPU capacity is doing nothing.” The best-measured cluster in the report, a 136-node H200 LLM inference fleet, reached 49% average utilization. The fleet-wide average is 5%. That 10x gap comes almost entirely from technique, not from hardware differences. GPU-Aware Bin-Packing Prerequisite: Right-Size GPU Requests Right-sizing resource requests is a prerequisite for every layer that follows. If requests are too large, nodes appear full while their GPUs sit idle. The scheduler allocates based on what pods declare, not what they consume. Use kubectl top pods --containers to identify pods consuming far less than they declare. For VRAM specifically, query DCGM FI DEV FB USED to see actual framebuffer consumption. Pods that declare a full GPU but consume 8 GB of an 80 GB H100 are prime candidates for MIG partitioning or time-slicing. Layer 1: Scheduler Scoring Switch the scheduler scoring plugin from LeastAllocated to MostAllocated. This tells the scheduler to prefer the most utilized node available, so new pods fill existing nodes before the cluster expands. The resources: field is required. Without it, the scheduler weighs CPU and memory equally with GPUs and GPU density does not drive placement. apiVersion: kubescheduler.config.k8s.io/v1 kind: KubeSchedulerConfiguration profiles: - schedulerName: default-scheduler pluginConfig: - name: NodeResourcesFit args: scoringStrategy: type: MostAllocated resources: - name: cpu weight: 1 - name: memory weight: 1 - name: nvidia.com/gpu weight: 3 The higher weight for nvidia.com/gpu ensures GPU bin-packing drives scheduling decisions. A node with a busy GPU scores higher than a node with busy CPU, which matches the economics of GPU infrastructure. Layer 2: Ongoing Consolidation Changing the scheduler fills new nodes more densely. It does not reclaim capacity from nodes already sitting underutilized. A consolidation controller handles that: it identifies underutilized nodes, migrates their pods, and removes empty nodes. Before enabling consolidation, ensure all production workloads have PodDisruptionBudgets configured. Cast AI Evictor and Karpenter both respect PDBs. A pod with no PDB can be evicted immediately during consolidation, terminating an in-progress training run without warning. A minimal PDB for an inference deployment: apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: inference-pdb spec: minAvailable: 1 selector: matchLabels: app: inference GPU node cold starts take 3 to 8 minutes on instances like AWS p4d and p5, including driver initialization and NVLink fabric bring-up. Aggressive consolidation that frequently terminates GPU nodes creates scheduling delays of that duration. Set consolidation thresholds to balance empty-node savings against cold-start cost. Cast AI Evictor pre-provisions replacement capacity before draining, reducing cold-start exposure. For Karpenter on EKS, the WhenUnderutilized policy works without a consolidateAfter field: disruption: consolidationPolicy: WhenUnderutilized budgets: - nodes: "10%" Note that consolidateAfter is only valid with the WhenEmpty policy. With WhenEmpty , Karpenter waits the specified duration after a node becomes fully empty before terminating it: disruption: consolidationPolicy: WhenEmpty consolidateAfter: 30m Using consolidateAfter with WhenUnderutilized is invalid and will cause the NodePool to be rejected at admission. For GPU fleets, WhenUnderutilized with a 10% node budget is the recommended starting point. For the complete consolidation approach, see the Kubernetes bin-packing guide https://cast.ai/blog/kubernetes-bin-packing/ . MIG: Hardware-Level Partitioning MIG Multi-Instance GPU partitions a single physical GPU into up to 7 isolated instances at the hardware level. Each MIG instance gets dedicated compute cores, dedicated memory bandwidth, and a dedicated L2 cache. Workloads in separate instances cannot interfere with each other. MIG is supported on A100, A30, H100, and H200. H100 MIG profiles, from smallest to largest: 1g.10gb : 7 instances, 10 GB each 2g.20gb : 3 instances, 20 GB each 3g.40gb : 2 instances, 40 GB each 7g.80gb : 1 instance, full GPU Time-Slicing: Software-Level Sharing Time-slicing creates virtual GPU replicas through CUDA context switching. Up to 48 virtual replicas per physical GPU are supported. It works on any CUDA-capable GPU, making it useful for older hardware that does not support MIG. For vLLM deployments, set --gpu-memory-utilization per replica. With 4 replicas sharing one GPU, use 0.22 rather than the arithmetic result of 0.25. The lower value reserves headroom for CUDA context overhead, which runs approximately 1 to 2 GB per context. Adjust the value based on your model’s actual VRAM footprint. Time-slicing has no memory enforcement boundary. Two pods each configured for 0.22 can simultaneously allocate VRAM, causing OOM failures that kill all co-tenants on the GPU. The --gpu-memory-utilization flag is an application-level soft limit only. It does not prevent another replica from allocating memory at the same time. Use MIG when memory isolation is a production requirement. MIG vs. Time-Slicing: When to Use Which | Criterion | MIG | Time-Slicing | |---|---|---| | Memory isolation | Hardware-enforced | None | | Noisy-neighbor risk | Eliminated | Present | | Hardware support | A100, A30, H100, H200 | Any CUDA GPU | | Fault isolation | Per-instance | None | | Mixed workload sizes | Yes | No | | Best for | Production multi-tenant serving | Dev environments, lightweight/stateless workloads | For detailed guidance on combining both approaches for maximum density, see the GPU sharing for Kubernetes cost optimization guide https://cast.ai/blog/gpu-sharing-kubernetes-cost-optimization/ . The Economics Four developers sharing one H100 at $6.88 per hour costs $1.72 per developer per hour. Add Spot pricing at a 60% discount and the cost drops to roughly $0.69 per developer per hour. That is approximately a 90% reduction compared to one dedicated H100 per developer. Despite this, fewer than 2% of enterprise Kubernetes GPU workloads currently run on Spot, even though AWS Spot saves 60 to 91%, GCP 60 to 80%, and Azure 60 to 90% compared to on-demand rates. Dynamic Resource Allocation What DRA Solves Dynamic Resource Allocation DRA is the Kubernetes API designed to replace the device plugin’s opaque integer model. DRA reached GA stable in Kubernetes 1.34 and is enabled by default in Kubernetes 1.35. AWS EKS supports DRA from Kubernetes 1.33. The API group is resource.k8s.io/v1 . DRA introduces four key objects: ResourceSlice : the driver publishes available device inventory to the API server DeviceClass : a cluster-wide policy defining a class of devices and their default configuration ResourceClaimTemplate : a per-namespace template for generating ResourceClaims ResourceClaim : the actual per-pod binding to a specific device Attribute-Based Scheduling with CEL With DRA, pods select devices using CEL Common Expression Language expressions rather than integer counts. CEL is a lightweight expression language embedded in Kubernetes that evaluates device attributes at scheduling time. Instead of asking for “one GPU,” a pod asks for a GPU with specific architecture and minimum memory. apiVersion: resource.k8s.io/v1 kind: ResourceClaim metadata: name: hopper-gpu-claim spec: devices: requests: - name: gpu deviceClassName: gpu.nvidia.com selectors: - cel: expression: device.attributes 'gpu.nvidia.com' .architecture == 'hopper' && device.attributes 'gpu.nvidia.com' .memory.isGreaterThan quantity '40Gi' This ResourceClaim selects only Hopper-architecture GPUs with more than 40 GB of memory. The scheduler evaluates the CEL expression against the ResourceSlice inventory and places the pod on a matching device. No more opaque counters. No more platform-blind scheduling. Default vs GPU-Aware vs DRA: Side-by-Side | Feature | Default LeastAllocated + Device Plugin | GPU-Aware Bin-Packing | DRA K8s 1.34+ | |---|---|---|---| | Scheduler scoring | LeastAllocated spread | MostAllocated pack | MostAllocated + attribute filtering | | GPU allocation model | Whole GPU, integer count | MIG or time-slicing aware | Attribute-based CEL matching | | Fractional sharing | No | Yes MIG, time-slicing | Yes | | Cross-pod device sharing | No | Partial | Yes | | GPU attribute matching | No | No | Yes | | Auto-consolidation | No | With Evictor or Karpenter | With Evictor or Karpenter | | Production-ready today | Yes | Yes | Yes K8s 1.34+ | DRA is not a drop-in replacement for all workloads today. Device plugins remain stable and widely deployed. For new clusters on Kubernetes 1.34 or later, however, DRA is the forward path. How Cast AI Automates GPU Scheduling and Bin-Packing Three Layers Plus One Manual implementation of the three technical layers requires configuration, ongoing maintenance, and cross-cluster coordination. MIG profiles change as workloads change. Scheduler scoring plugins need deployment and monitoring. Consolidation needs active eviction logic with cold-start awareness built in. At 50 nodes, this is manageable. At 500 nodes, it is a full-time job. Cast AI automates all three layers and adds a fourth. The three technical layers are: MostAllocated scheduler scoring with GPU-weighted resources deployed cluster-wide; GPU sharing via MIG partition management and time-slicing replica configuration; and continuous consolidation through the Cast AI Evictor, which verifies spare capacity using actual DCGM metrics rather than request declarations, migrates pods while respecting PodDisruptionBudgets, and removes empty nodes. The fourth layer is automated GPU node lifecycle: scale-to-zero for idle GPU node pools, pre-provisioning replacement capacity before draining nodes to avoid cold-start delays, and predictive scaling based on workload patterns. This runs across AWS, GCP, and Azure without separate per-cloud configuration. ALLEN Digital combined time-slicing, Spot instances, and Cast AI bin-packing consolidation to achieve over 70% total savings compared to their prior SageMaker setup. That result comes from stacking all four layers, not from applying any single one in isolation. An idle H100 costs roughly $4,954 per month. Recovering even one idle GPU per week across a mid-size fleet adds up fast. For the complete GPU optimization framework, including DCGM alerting thresholds and MIG configuration templates, see the Kubernetes GPU optimization guide https://cast.ai/blog/kubernetes-gpu-optimization/ . Conclusion GPU scheduling and bin-packing are not configuration details. They are the operational difference between 5% utilization and the 49% achievable with the right techniques. The default scheduler spreads pods thin. The device plugin cannot see GPU attributes. Neither problem requires new hardware to solve. Fixing it requires three technical layers. First, switch scheduler scoring to MostAllocated with GPU-weighted resources. Second, add GPU sharing via MIG or time-slicing, choosing based on whether memory isolation is required. Third, run continuous consolidation with PDB protection and cold-start awareness in place. Cast AI adds the fourth layer: automated GPU node lifecycle that removes the operational burden of managing cold-start windows, pre-provisioning, and scale-to-zero across multiple clouds. Each step recovers real dollars at real GPU prices. For the full Kubernetes GPU optimization framework, start with the Kubernetes GPU optimization guide https://cast.ai/blog/kubernetes-gpu-optimization/ . Frequently Asked Questions How does Kubernetes schedule GPUs? Kubernetes schedules GPUs using the device plugin framework. Vendors deploy DaemonSet-based plugins that advertise GPUs as extended resources, such as nvidia.com/gpu . Pods request a quantity of that resource, and the scheduler finds a node with enough available units. Allocation is whole-GPU by default: one request equals one full physical device, regardless of actual usage. The scheduler treats GPUs as opaque integer counters with no attribute awareness. What is Dynamic Resource Allocation? Dynamic Resource Allocation DRA is a Kubernetes API that replaces the device plugin’s integer count model with attribute-based device selection. Pods use ResourceClaims with CEL expressions to specify exactly which device attributes they require, for example GPU architecture or minimum memory. DRA reached stable status in Kubernetes 1.34, is enabled by default in Kubernetes 1.35, and supports device sharing across pods through centralized DeviceClass objects. How do you improve GPU bin-packing in Kubernetes? GPU bin-packing improvement requires three steps. First, right-size GPU resource requests so nodes do not appear full while their GPUs sit idle. Use kubectl top pods --containers and DCGM FI DEV FB USED to measure actual consumption. Second, switch scheduler scoring from LeastAllocated to MostAllocated with a higher weight for nvidia.com/gpu so new pods land on the most utilized node first. Third, add ongoing consolidation with PodDisruptionBudgets on all workloads and cold-start thresholds set to avoid excessive GPU node churn. Cast AI’s Evictor and Karpenter both support this consolidation pattern. What is the NVIDIA device plugin? The NVIDIA device plugin is a DaemonSet that runs on each GPU node and registers available NVIDIA GPUs with the Kubernetes kubelet. It exposes GPUs as the extended resource nvidia.com/gpu . The NVIDIA GPU Operator automates deployment of the device plugin alongside GPU drivers, the DCGM Exporter for metrics, the MIG Manager DaemonSet, and GPU Feature Discovery for node labeling. You install the GPU Operator via a single Helm chart.