Fractional GPUs and GPU Rightsizing: Stop Wasting Whole Cards

Average GPU utilization in production Kubernetes clusters is just 5%, wasting 95% of GPU spend, according to the Cast AI 2026 State of Kubernetes Optimization Report. Fractional GPUs via MIG or time-slicing, combined with GPU rightsizing, can raise utilization to 49% without buying more hardware. Cast AI automates the rightsizing loop to eliminate idle GPU capacity.

A fractional GPU lets a workload use part of a physical GPU instead of holding the whole card, using MIG partitions or time-slicing. Combined with GPU rightsizing matching GPU and memory requests to real usage , it is the most direct way to raise GPU utilization from the typical 5% without buying more hardware. Key takeaways - Average GPU utilization in production Kubernetes clusters is just 5%, according to the Cast AI 2026 State of Kubernetes Optimization Report https://cast.ai/reports/state-of-kubernetes-optimization/ . - Fractional GPUs, via MIG or time-slicing, let multiple pods share one physical GPU card. - MIG provides hardware-level memory and fault isolation. Time-slicing is software-only and works on any NVIDIA GPU. - GPU rightsizing matches resource requests to actual utilization, eliminating waste on capacity workloads never consume. - At 4 time-slicing replicas, each workload’s allocated GPU share drops to 25% of the card. - A single H100 or H200 supports up to 7 MIG instances. - Cast AI automates the full rightsizing loop: measure utilization, generate recommendations, and apply corrected requests automatically. Why whole-GPU allocation wastes money Kubernetes GPU scheduling is all-or-nothing by default. A pod requests nvidia.com/gpu: 1 and the scheduler pins the entire physical card to that pod. No other workload can touch it, even when the pod is idle. In production, most inference pods consume a tiny fraction of the GPU they hold. Average utilization across production clusters is just 5%, based on data from tens of thousands of clusters on AWS, GCP, and Azure. That figure comes directly from the Cast AI 2026 State of Kubernetes Optimization Report https://cast.ai/reports/state-of-kubernetes-optimization/ . In other words, 95 cents of every GPU dollar goes to idle silicon. At AWS Capacity Block pricing for H100 instances ~$6.88 per GPU-hour , a 20-GPU cluster running 720 hours per month at 5% utilization wastes roughly $94,000 per month in idle GPU capacity. The economics are getting harder, not easier. H200 AWS Capacity Block pricing increased 15% in January 2026, the first GPU price increase in roughly two decades, according to the Cast AI 2026 State of Kubernetes Optimization Report https://cast.ai/reports/state-of-kubernetes-optimization/ . Teams scaling AI workloads pay more per card while using less of each card. That is a compounding problem with no self-correcting mechanism. The gap is technique, not hardware. The best-performing cluster in the Cast AI dataset is a 136-node H200 fleet running at 49% utilization, roughly 10x the average. Those teams apply fractional GPU allocation and rightsizing. The hardware is the same; the configuration is different. Fractional GPUs: MIG vs. time-slicing For a broader look at GPU sharing strategies, see the Cast AI guide on GPU sharing in Kubernetes for cost optimization https://cast.ai/blog/gpu-sharing-kubernetes-cost-optimization/ . Two mechanisms enable GPU sharing in Kubernetes: Multi-Instance GPU MIG and time-slicing. Both work through the NVIDIA GPU Operator. They differ in isolation guarantees, supported hardware, and Kubernetes resource naming. MIG: hardware-level partitions MIG is a hardware feature available on NVIDIA A100, H100, and H200 GPUs. It partitions the physical GPU into isolated slices at the silicon level. Each slice gets dedicated memory, compute engines, and cache. One slice cannot access another’s memory, and a crash in one instance does not affect others. A single H100 80GB supports up to 7 MIG instances using the 1g.10gb profile, giving each slice 10 GB of framebuffer memory. The H200 141GB supports 7 instances at 1g.18gb , providing 18 GB per slice. The table below covers profiles across GPU generations: | GPU | Memory | Smallest profile | Max instances | |---|---|---|---| | A100 40GB | 40 GB | 1g.5gb | 7 | | A100 80GB | 80 GB | 1g.10gb | 7 | | H100 80GB | 80 GB | 1g.10gb | 7 | | H200 141GB | 141 GB | 1g.18gb | 7 | The following ConfigMap enables 7x 1g.10gb slices on all H100 80GB devices via the NVIDIA GPU Operator: apiVersion: v1 kind: ConfigMap metadata: name: default-mig-parted-config namespace: gpu-operator data: config.yaml: | version: v1 mig-configs: all-1g.10gb: - devices: all mig-enabled: true mig-devices: "1g.10gb": 7 Apply the ConfigMap kubectl apply -f mig-parted-config.yaml Before labeling, verify the node runs an MIG-capable GPU: kubectl get node <gpu-node -o jsonpath='{.metadata.labels.nvidia\.com/gpu\.product}' Only A100, H100, and H200 GPUs support MIG. Applying the label to a T4 or A10 node will fail without error, leaving MIG unconfigured. Label the node s to trigger MIG Manager reconfiguration kubectl label node <gpu-node nvidia.com/mig.config=all-1g.10gb --overwrite MIG reconfiguration requires draining the node first for in-flight workloads. The NVIDIA MIG Manager handles the actual device reconfiguration. Verify MIG instances were created nvidia-smi mig -lgip Or check Kubernetes sees them kubectl describe node <gpu-node | grep mig Pods request a MIG slice using the specific resource name: resources: limits: nvidia.com/mig-1g.10gb: "1" Time-slicing: software-level multiplexing Time-slicing works on any NVIDIA GPU, including older models like T4 and A10. The driver multiplexes GPU access across multiple pods in rapid alternation. There is no memory isolation between pods. A runaway workload can consume all available VRAM. Time-slicing is far simpler to configure and requires no special hardware support. The following ConfigMap sets 4 replicas per GPU, giving each pod a 25% time share: apiVersion: v1 kind: ConfigMap metadata: name: time-slicing-config namespace: gpu-operator data: any: |- version: v1 flags: migStrategy: none sharing: timeSlicing: resources: - name: nvidia.com/gpu replicas: 4 Apply the ConfigMap kubectl apply -f time-slicing-config.yaml Patch the ClusterPolicy to activate the config kubectl patch clusterpolicies.nvidia.com/cluster-policy \ -n gpu-operator --type merge \ -p '{"spec": {"devicePlugin": {"config": {"name": "time-slicing-config", "default": "any"}}}}' Verify replicas are exposed kubectl describe node <gpu-node | grep nvidia.com/gpu At 4 replicas, each workload’s allocated GPU share is 25% of the card. If actual compute demand is spread across all 4 pods simultaneously, effective per-workload GPU cost drops by roughly 75%. If all pods run at peak simultaneously, they compete for the same time slots, but idle inference pods still benefit fully from the cost sharing. MIG vs. time-slicing: direct comparison | Feature | MIG | Time-Slicing | |---|---|---| | Memory isolation | Yes hardware | No | | Fault isolation | Yes hardware | No | | Kubernetes support | nvidia.com/mig-Xg.Ygb resource name | nvidia.com/gpu with replica count | | GPU models supported | A100, H100, H200 | Any NVIDIA GPU | | Max instances per GPU | 7 | Configurable typically 4-8 | | Best for | Multi-tenant inference, strict isolation | Dev/test, batch, cost-sensitive inference | A third option, NVIDIA Multi-Process Service MPS , allows concurrent CUDA kernel execution on the same GPU with some memory protection. MPS is suitable for trusted, same-tenant workloads where strict MIG isolation is not required. GPU rightsizing: match requests to real usage Fractional GPUs reduce allocation granularity. Rightsizing ensures your requests actually match workload behavior. Together, they eliminate two distinct kinds of waste: holding a whole card when you need a fraction, and requesting a large fraction when you need a small one. Measuring with DCGM metrics The NVIDIA DCGM Exporter exposes the metrics you need. Three are essential for rightsizing decisions: DCGM FI DEV GPU UTIL GPU compute utilization % DCGM FI DEV FB USED Framebuffer memory used MiB DCGM FI DEV FB FREE Framebuffer memory free MiB Use these PromQL queries to establish the utilization baselines you need for rightsizing decisions: 7-day average GPU compute utilization 5-minute resolution steps avg over time DCGM FI DEV GPU UTIL 7d:5m Peak GPU memory used over 7 days max over time DCGM FI DEV FB USED 7d:5m Collect at least 7 days of data, spanning your typical traffic peaks, before drawing rightsizing conclusions. Then compare observed peaks against what each pod’s resource requests specify. In most production clusters, pods request a full GPU but peak at 5-10% compute utilization and use 20-30% of GPU memory. That discrepancy is the rightsizing opportunity: most pods could run on a single MIG slice and still have headroom. Setting corrected resource requests After measuring, set GPU requests to match observed peak usage with a reasonable buffer. For memory, base your request on the P95 of DCGM FI DEV FB USED plus 20% headroom. For compute, GPU requests in Kubernetes are integers, so rightsizing often means switching to a smaller MIG profile rather than tuning a fractional value. One important caveat: in-place pod resizing for CPU and memory entered beta in Kubernetes 1.33. GPU resource changes still require a pod restart. Plan rightsizing during deployment windows or maintenance cycles. How Cast AI automates GPU rightsizing Manual rightsizing across hundreds of workloads is operationally expensive. The Cast AI Workload Autoscaler integrates DCGM Exporter metrics via the Kvisor agent. It continuously measures real GPU and memory utilization, generates rightsizing recommendations, and applies them automatically without manual intervention. After rightsizing, PrecisionPack bins workloads tightly onto fewer nodes, compounding the efficiency gains. Teams using Cast AI for automated workload rightsizing https://cast.ai/blog/automated-workload-rightsizing-precisionpack/ have moved GPU utilization from the 5% cluster average into the 40–60% range across production clusters, based on Cast AI telemetry from its 2026 State of Kubernetes Optimization dataset. For the full picture on Kubernetes GPU optimization https://cast.ai/blog/kubernetes-gpu-optimization/ , the Cast AI blog covers the complete stack from scheduling to bin-packing. When to use which: inference vs. training Choosing between MIG, time-slicing, and rightsizing depends on workload type and isolation requirements. The following table provides direct guidance: | Workload | Recommended approach | Reason | |---|---|---| | Latency-sensitive inference | MIG | Predictable memory, no noisy-neighbor effects | | Multi-tenant inference | MIG | Hardware fault and memory isolation per tenant | | Batch inference | Time-slicing | Simpler config, lower cost per job | | Dev/test environments | Time-slicing | Works on any GPU, fast to configure | | Large model training | Single GPU, no sharing | Training runs need full bandwidth and memory | | Distributed training | Single GPU per rank, no sharing | Cross-GPU communication latency is critical | | All workloads | Rightsizing | Most clusters have more than 50% memory headroom | Pick your approach from the table, then layer rightsizing on top regardless of which sharing method you choose. Rightsizing applies universally; fractional allocation depends on your isolation requirements and GPU hardware generation. Conclusion If your cluster runs A100, H100, or H200 nodes, start with MIG. It gives you hardware-level isolation and makes fractional GPU allocation safe for production multi-tenant inference. If your hardware is older, time-slicing gets you fractional allocation immediately, without waiting for your next hardware refresh. In both cases, begin with rightsizing. It requires no hardware constraints and delivers savings on day one. Measure DCGM utilization for at least 7 days, set corrected requests, then layer fractional allocation based on workload isolation needs. Tighter requests plus shared physical GPUs compound together: that combination is where the largest efficiency gains appear. Cast AI handles the rightsizing loop automatically. The Workload Autoscaler monitors GPU utilization via DCGM, recommends corrected requests, and applies them without manual intervention. PrecisionPack then bins the rightsized workloads tightly, multiplying the effect across your fleet. For production benchmark data, start with the Cast AI 2026 State of Kubernetes Optimization Report https://cast.ai/reports/state-of-kubernetes-optimization/ . To start: connect your DCGM metrics to Cast AI’s Workload Autoscaler, let it observe 7 days of utilization data, and run your first automated rightsizing cycle. Most teams see their first GPU request correction within 48 hours. See how automated GPU optimization can help you turn GPU utilization data into lower infrastructure costs. https://cast.ai/gpu-optimization Frequently Asked Questions What is a fractional GPU? A fractional GPU is a portion of a physical GPU card assigned to a single workload. Instead of one pod holding an entire GPU, multiple pods each use a defined share of the card. MIG creates fixed hardware partitions with isolated memory and fault boundaries. Time-slicing multiplexes GPU compute time across pods in software, without memory separation between workloads. Can multiple pods share one GPU? Yes. Both MIG and time-slicing allow multiple pods to share one physical GPU. With MIG, each pod receives a dedicated hardware slice with isolated memory and independent fault containment. With time-slicing, pods take turns using the GPU at the driver level and share the same memory pool without isolation between workloads. MIG vs. time-slicing: which is better for fractional GPUs? MIG is better when isolation matters. Each MIG instance has dedicated memory and fault containment at the hardware level. Time-slicing is simpler and works on any NVIDIA GPU, but provides no memory isolation between pods. For production multi-tenant inference, use MIG. For dev/test or batch workloads on older hardware, time-slicing is generally sufficient and far easier to configure. How do you rightsize GPU requests in Kubernetes? Collect DCGM metrics, specifically DCGM FI DEV GPU UTIL and DCGM FI DEV FB USED , over at least 7 days to measure peak compute and memory utilization. Compare observed peaks against current requests and limits. Then adjust requests to match real usage plus a headroom buffer of roughly 20% for memory. Note that GPU resource changes require a pod restart in current Kubernetes versions unlike CPU and memory, which support in-place resizing in beta as of K8s 1.33 . Cast AI automates this entire process: the Workload Autoscaler continuously measures utilization and applies corrected requests without manual intervention.