The GPU Shortage Inside Your Own Infrastructure: Why AI Workloads Queue While Capacity Sits Idle

Cast AI's 2026 Kubernetes Optimization Report, analyzing tens of thousands of production clusters from January 2025 through April 2026, recorded average GPU compute utilization of 5% before any optimization was applied, with one cluster sustaining 49% utilization across 136 H200s. The report attributes the gap almost entirely to technique rather than hardware, noting that low compute utilization does not mean a GPU is available, since Kubernetes assigns whole GPUs to pods by default and a device holding a loaded model remains allocated regardless of how little compute it uses. Cast AI supports time-slicing, MIG, and MPS sharing methods with automatic bin-packing and no changes required to workload manifests.

AI workloads can wait for GPUs even when monitoring shows idle compute. A GPU with low compute activity is not necessarily available capacity for another workload because allocation rules, memory requirements, hardware compatibility, isolation, and placement constraints determine what can actually run. The post should explain how to distinguish an actual hardware shortage from an allocation, placement, or application bottleneck. Use the 5% average GPU utilization finding from Cast AI’s 2026 report as a research hook, but do not imply that queued workloads and idle GPUs occurred together in the same measured environments or that 95% of GPU capacity is immediately reclaimable. Key takeaways - GPU utilization metrics measure compute activity, not allocation state. A GPU reporting 5% compute utilization can still be fully allocated with zero capacity available for new workloads. - Kubernetes assigns whole GPUs to pods by default. One workload holding a device blocks all others regardless of how much of the GPU it actually uses. - Queued workloads beside idle-looking GPUs have at least four distinct root causes. Only one of them is resolved by GPU sharing. - Time-slicing, MIG, and MPS trade off differently on memory isolation, hardware requirements, observability, and cloud provider support. No single method fits every workload. - The diagnostic sequence matters: check allocation state, then memory occupancy, then placement constraints, then application bottlenecks. In that order. - Cast AI supports all three sharing methods with automatic bin-packing and no changes required to workload manifests. What GPU utilization tells you and what it misses GPU utilization is a phrase that covers at least four different measurements, and conflating them is the fastest route to a misdiagnosis. What most dashboards surface is compute utilization: the percentage of streaming multiprocessors SMs that are active in a given time window. That number tells you how busy the silicon is when it runs. It tells you nothing about whether the device is available for a new workload. A model loaded into VRAM occupies that memory continuously. The inference service may answer one request per minute, keeping compute activity at 5%, but the device is allocated, the memory is occupied, and Kubernetes will not schedule anything else on it. From the scheduler’s perspective, that GPU is unavailable. From your monitoring dashboard, it looks nearly idle. The table below separates four metrics commonly grouped under “GPU utilization” and clarifies what each one does and does not tell you. | Metric | What it measures | What it does NOT show | |---|---|---| | Compute activity %SM | Fraction of streaming multiprocessors active during the sample window | Whether the GPU is allocated; whether VRAM is available for another workload | | Memory occupancy VRAM | How much GPU memory is consumed by loaded models and tensors | Compute activity; whether queued workloads could fit in remaining memory | | Queue wait time pod scheduling | How long pending pods wait before a GPU device becomes available | Whether the delay is caused by allocation, memory, placement, or application bottleneck | | Throughput / latency | Requests served per second; time-to-first-token or end-to-end response time | GPU resource utilization or allocation efficiency | The Cast AI 2026 Kubernetes Optimization Report https://cast.ai/reports/state-of-kubernetes-optimization/ , which analyzed tens of thousands of production clusters from January 2025 through April 2026, recorded average GPU compute utilization of 5% before any optimization was applied. The accompanying report blog post https://cast.ai/blog/2026-state-of-kubernetes-resource-optimization-cpu-at-8-memory-at-20-and-getting-worse/ notes that one cluster in the dataset sustained 49% GPU utilization across 136 H200s, a gap described as almost entirely attributable to technique rather than hardware. That 5% average reflects a real fleet-wide pattern of overprovisioning and underuse. It does not mean 95% of capacity is free to accept new workloads. Each cluster, and each GPU within it, needs its own diagnostic pass before that conclusion holds. Four reasons workloads can queue beside idle GPU capacity The Kubernetes scheduler sees a GPU as either available or unavailable. It makes that determination based on the nvidia.com/gpu resource count on each node, not on what percentage of the silicon is active. Four distinct conditions can produce a queue even when compute activity looks low. A workload holds an entire device The NVIDIA Kubernetes Device Plugin allocates GPUs exclusively by default. When a pod requests nvidia.com/gpu: 1 , it receives sole ownership of one physical device for the duration of its lifecycle. No other pod can use that device, regardless of how much compute or memory the occupying workload actually consumes. This is the most common cause of the queue-beside-idle pattern in inference clusters. A set of models, each holding a dedicated GPU but serving bursty or low-frequency requests, keeps every device allocated. An incoming workload finds nvidia.com/gpu: 0 available on the node and waits, even though aggregate compute activity across the node might be under 10%. The device plugin focuses on allocation, not reclamation. Kubernetes cluster autoscaler can provision new nodes but will not recover idle capacity on existing ones. The problem lives at the scheduling layer, and the solution requires changing how devices are presented to the scheduler. That is exactly what GPU sharing mechanisms do. Memory is occupied, not just underused Low compute utilization does not mean memory is available. Two scenarios illustrate the range. A 70B-parameter model loaded in FP16 consumes roughly 140 GB of VRAM base weights only; at 4K context lengths, KV cache adds 15–20% above base weights, while at 128K context lengths, KV cache requirements can exceed base weights entirely . That model requires two or more A100 80GB GPUs to load at all. A 13B FP16 model occupies roughly 26 GB and fits on a single A100 80GB, but it still holds that VRAM continuously. Compute might report 8%, because most requests complete quickly, but the device cannot accept another workload. Before configuring sharing, check whether quantization applies. An INT4-quantized 7B model occupies roughly 4 GB versus roughly 14 GB in FP16, which changes the co-location arithmetic for time-slicing and MIG configurations. INT8 cuts requirements roughly in half relative to FP16, often with acceptable quality trade-offs for inference workloads. This matters particularly for teams considering time-slicing as a solution. Time-slicing multiplexes access to compute, but it does not partition memory. All replicas share the same VRAM address space. If two workloads together exceed device memory, they will not coexist safely. Diagnosing a memory constraint first, before choosing a sharing method, prevents a failed deployment and potential workload instability. Check memory occupancy with nvidia-smi before assuming a sharing approach will help. If the free memory column is near zero on the relevant devices, the constraint is VRAM capacity, not compute scheduling. MIG provides hardware-isolated memory partitions that can address certain versions of this problem, but it requires compatible hardware and changes how workloads are scheduled. Placement rules exclude otherwise usable hardware A workload may fail to schedule not because GPUs are fully allocated but because the scheduler cannot find a node that satisfies all placement constraints simultaneously. Node affinity rules specifying a GPU model or generation, topology spread constraints requiring a minimum number of nodes, and taints without matching tolerations can all prevent scheduling even when raw device counts look sufficient. A request for nvidia.com/gpu: 1 with a node selector requiring an A100 will not land on a node carrying only H100s, even if those H100s are idle and capable. Similarly, a workload requesting more time-sliced replicas than any single node provides, without spreading across nodes, can fail entirely despite the fleet having aggregate capacity. The diagnostic command here is kubectl describe pod <pending-pod . Look at the Events section for reasons like MatchNodeSelector , InsufficientResource , or TaintToleration . Placement failures often look like capacity shortages, and they resolve through scheduling configuration, not hardware procurement. The running workload is waiting on something else Not every low compute reading signals an allocation problem. GPU compute sits idle during CPU preprocessing, data loading, PCIe transfers, and network round trips. An LLM inference server waiting on a slow tokenization step or a KV cache fetch from remote storage will show low %SM activity, but the GPU is not available for another workload’s inference requests. It is blocked at a different layer. This category is the hardest to diagnose from GPU metrics alone. Sustained low compute on an active, scheduled workload usually points to a bottleneck outside the GPU. Increasing GPU count will not help. Fixes involve CPU resource increases, batching configuration adjustments, storage latency improvements, or data pipeline redesign. Use nvidia-smi dmon or DCGM metrics to observe compute utilization over time. A workload cycling between 0% and high compute in short bursts is likely CPU-gated during the 0% periods. A workload stuck at persistently low compute while actively receiving requests is waiting on I/O or network. Both patterns are distinct from an allocation shortage and require different remedies. A worked Kubernetes GPU allocation example The scenario below is hypothetical and constructed to illustrate the allocation problem. It does not represent measured performance from any specific cluster. Consider a node with four GPUs, each allocated exclusively to one inference service. Each service handles bursty student queries, with peak compute activity of 15% during busy periods and under 3% during off-hours. A fifth inference service deploys and enters Pending state. The scheduler finds nvidia.com/gpu: 0 available on the node and queues the pod. | Configuration | GPUs on node | Services running | Peak compute activity | Fifth service status | |---|---|---|---|---| | Current: dedicated allocation | 4 | 4 one per GPU | ~15% per device | Pending: no devices available | | Potential: time-sliced 4 replicas each | 4 | Up to 16 four per GPU | Varies by request overlap | Schedulable, subject to VRAM fit | Shifting to time-slicing does not guarantee any specific throughput improvement. What it changes is device availability from the scheduler’s perspective. Each GPU now advertises four slots instead of one. The fifth service can schedule. Whether total latency improves or degrades depends on how often the services’ requests overlap, how much VRAM each model occupies, and whether any model exhibits bursty SM activity that creates contention during peak windows. The only way to know the answer for a specific workload mix is to test it. The diagnostic steps that follow describe how to do that without committing to a configuration that might harm production latency. Dedicated GPUs, time-slicing, MIG, and MPS compared Four allocation models exist for GPU workloads in Kubernetes. Each solves a different version of the sharing problem, and each carries constraints that determine whether it fits a given workload. No single approach is universally correct. | Approach | Allocation model | Memory isolation | Hardware requirements | Workload fit | Key tradeoff | |---|---|---|---|---|---| | Dedicated allocation | One pod, one GPU, exclusive access | Full isolation | Any NVIDIA GPU | Large models; multi-GPU training; latency-critical inference needing full device bandwidth | Maximum isolation; zero sharing; wastes capacity for intermittent or low-utilization workloads | | Time-slicing | Temporal multiplexing; N replicas share one physical GPU | None: all replicas share the full VRAM address space | Any NVIDIA GPU; works on V100, T4, and older generations | Small models; intermittent inference; batch jobs with variable demand | No memory or fault isolation; DCGM-Exporter cannot attribute metrics to containers; VRAM contention risk under concurrent load | | MIG Multi-Instance GPU | Hardware partitioning into up to 7 isolated instances per physical GPU | Hardware-isolated memory paths: separate L2 cache banks, memory controllers, and DRAM buses per instance | Ampere architecture and newer only: A100, A30, H100, H200, Blackwell B200, GB200 . Not available on T4, V100, or A10. | Multi-tenant inference; workloads requiring QoS guarantees; mixed-size model serving | Fixed profiles, not dynamically resizable; NVLink not supported between instances; requires MIG mode enable and GPU reset | | MPS Multi-Process Service | Concurrent CUDA process execution; multiple processes share SMs simultaneously | No memory isolation in basic configuration. MPS with SM partitioning available from CUDA 11.0 onward adds configurable SM and memory partitions per namespace; Volta and newer add process-level fault isolation. Physical memory remains shared across all configurations. | Any CUDA-capable NVIDIA GPU. Cast AI MPS is currently available on GCP GKE; AWS and Azure support is on the roadmap. | Co-located workloads that can share execution; lower-isolation batch inference | Concurrent execution, not temporal interleaving; reduced context-switch overhead; evaluate MIG where strict per-workload memory budgets are required | Time-slicing works on any NVIDIA GPU, including older generations like T4 and V100 that do not support MIG. The NVIDIA GPU Operator time-slicing documentation https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html states explicitly that there is no memory or fault isolation between replicas. A workload that exhausts VRAM or crashes the CUDA context can affect all co-scheduled workloads on the same device. That constraint limits time-slicing to workloads whose combined memory footprint fits comfortably within device VRAM. MIG provides the strongest isolation guarantee of the three sharing methods. Each partition has physically separate paths through the memory system, including on-chip crossbar ports, L2 cache banks, memory controllers, and DRAM address buses. One instance’s cache thrashing cannot affect another’s latency. The NVIDIA MIG User Guide https://docs.nvidia.com/datacenter/tesla/mig-user-guide/introduction.html documents up to seven instances per physical GPU, with fixed profiles that define compute and memory allocation per partition. Those profiles are not dynamically resizable at runtime, so the partitioning decision is made at configuration time. One constraint matters specifically for distributed training workflows: NVLink is not supported between MIG instances. Teams running multi-GPU training jobs that rely on high-bandwidth inter-GPU communication should not treat MIG as a drop-in sharing option for those workloads. MPS is distinct from both. Where time-slicing switches between CUDA contexts over time, MPS enables multiple CUDA processes to execute concurrently on the same streaming multiprocessors. The NVIDIA MPS documentation https://docs.nvidia.com/deploy/mps/latest/index.html describes the architecture as a lightweight runtime: a control daemon, a server process, and a client library built into libcuda.so. MPS with SM partitioning available from CUDA 11.0 onward adds configurable SM and memory partitions per namespace, giving administrators more isolation than basic time-slicing without requiring MIG-compatible hardware. Volta architecture and newer add process-level fault isolation; physical memory remains shared across all MPS configurations. Cast AI supports combining MIG and time-slicing. A single A100 partitioned into seven MIG instances, each configured with four time-sliced replicas, presents 28 logical GPU slots to the Kubernetes scheduler from one physical device. That is a high-density configuration that requires careful workload profiling before deployment, but the arithmetic illustrates why sharing method selection matters as much as hardware procurement. Diagnose the bottleneck before ordering more GPUs The sequence below works through the four bottleneck categories in order of least to most complex. Stop when you find the active constraint. The remaining steps only add noise. Step 1: Check allocation state across nodes. Run: kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu' If all nodes show 0 available GPUs and workloads are queued, you have an allocation shortage. GPU sharing is the relevant solution class. If some nodes show available GPU slots and workloads are still queued, move to step 3 to check placement constraints. Step 2: Check memory occupancy on allocated devices. Run nvidia-smi on nodes where compute activity is low but devices are fully allocated. Compare the used memory column against total memory for each device. Low %SM combined with near-zero free memory means the constraint is VRAM, not compute. Time-slicing will not help here. Memory partitioning via MIG, or MPS with SM partitioning, is the relevant path, subject to hardware and provider compatibility. Step 3: Inspect pending pod events for placement failures. Run: kubectl describe pod <pending-pod-name Look for scheduling failure reasons in the Events section: MatchNodeSelector , InsufficientResource , or TaintToleration . If placement rules are blocking the workload from landing on available GPUs, the fix is scheduling configuration. Hardware procurement will not resolve it. Step 4: Observe compute activity over time on running workloads. Use nvidia-smi dmon -s u or DCGM metrics to collect a time-series sample across a representative request period. If %SM cycles between 0 and high values in short bursts, the workload is likely CPU-gated during the 0% periods. If %SM stays persistently low while the workload actively receives requests, suspect data loading, PCIe transfer, or KV cache latency. Both patterns are distinct from an allocation shortage. Step 5: Verify hardware and provider compatibility for your intended sharing method. Time-slicing works on any NVIDIA GPU. MIG requires Ampere architecture or newer. Cast AI MPS runs on GCP GKE; see the comparison table above for roadmap status. Choosing a sharing method that the hardware or cloud provider does not support produces a configuration error before it produces a workload, so this check belongs before any configuration change. One observability constraint worth noting before finalizing a time-slicing configuration: DCGM-Exporter cannot associate per-container GPU metrics when time-slicing is enabled. Per-workload GPU visibility disappears unless you have a separate instrumentation path. Plan for that monitoring gap before the configuration reaches production, not after. When DCGM cannot attribute metrics to containers under time-slicing, use inference server native telemetry. vLLM exposes per-request GPU memory usage and queue depth at the /metrics endpoint. TGI provides similar metrics. How to test GPU sharing without sacrificing service quality The risk in enabling GPU sharing on a production cluster is concrete. Time-slicing without memory isolation can cause latency spikes when co-scheduled workloads submit burst requests simultaneously. A test-before-commit approach reduces that risk to something manageable. Start with a single node. Enable time-slicing on one node in a non-production namespace using a ConfigMap applied to the NVIDIA GPU Operator. Set the replica count to match the number of workloads you intend to co-schedule, beginning at two and increasing from there. Run each workload at realistic request rates before adding the next one. Measure latency at p95 and p99 percentiles throughout the ramp, not just the mean. Time-slicing contention surfaces at the tail first. If p99 remains within your service-level objective as you add workloads, the configuration is viable for that workload mix. If it degrades beyond tolerance before you reach your target replica count, time-slicing is not the right method for those specific workloads at those request rates. For workloads with strict latency budgets, evaluate MIG before committing to time-slicing. MIG’s hardware-isolated memory paths eliminate the cross-workload interference that time-slicing cannot prevent. On an A100, a 3g.20gb MIG profile provides 3/7 of the compute and approximately 20 GB of isolated VRAM, with deterministic throughput regardless of what runs in adjacent partitions. The tradeoff is hardware compatibility: MIG requires Ampere or newer. If your workloads run on GCP GKE, MPS is another option worth evaluating for concurrent inference scenarios see the comparison table for availability and isolation details . One factor to account for before running any co-location test: vLLM pre-allocates 90% of device VRAM at CUDA context initialization by default, regardless of request load. TGI pre-allocates based on the --max-batch-prefill-tokens setting, which can claim a substantial portion of VRAM before any inference request is served. An idle vLLM instance holds that 90% even when serving zero requests. vLLM exposes a --gpu-memory-utilization flag that caps this allocation; the default is 0.9. For time-slicing configurations where multiple inference servers share a device, set this flag lower on each co-resident pod to leave headroom for neighbors. Without this adjustment, two idle-looking inference servers can saturate VRAM before a single inference request arrives. Keep the DCGM monitoring gap in mind during testing. Per-container GPU metrics are not available through DCGM-Exporter with time-slicing enabled. Use application-level metrics during the evaluation: request latency, tokens per second, and queue depth. Plan for the monitoring limitation before moving the configuration to production. If time-slicing causes OOM events during testing, revert the GPU replicas count to 1 in the ConfigMap which disables time-slicing and restart the affected pods before adjusting workload VRAM budgets. Confirm VRAM fit before enabling any sharing configuration. Sum the VRAM requirements of all workloads you plan to co-schedule on a single device. If the total exceeds device memory, the configuration produces out-of-memory failures under load, not graceful degradation. MIG addresses this by assigning fixed memory per partition. Time-slicing does not. Running these tests manually is practical for one or two clusters. As GPU fleets grow, the same coordination logic benefits from automation. Where GPU sharing and placement automation fit A sharing configuration tuned for today’s workload mix degrades as soon as new models ship. Request rates shift. Teams add workloads without touching the ConfigMap from six months ago. Manual configuration can fix an allocation problem once; it cannot keep up with a fleet. Cast AI’s GPU sharing https://cast.ai/gpu-sharing/ addresses the exclusive allocation problem described above at the automation layer. Sharing configuration lives in node templates rather than workload manifests, so existing pods move to shared devices without manifest changes. Time-slicing scales from 1 to 48 replicas per GPU across any NVIDIA GPU: the same scenario from the worked example four inference services holding four dedicated devices at 10% compute resolves by raising the replica count, without rewriting a single Deployment spec. The placement constraint described in “Placement rules exclude otherwise usable hardware” is where bin-packing matters. Cast AI’s placement automation selects sharing configuration and distributes workloads across shared and partitioned GPUs without requiring node selectors or affinity rules in application manifests. Device assignment runs through Dynamic Resource Allocation DRA, stable in Kubernetes 1.32; earlier versions require feature gate enablement . DRA replaces the Extended Resources mechanism for device allocation, giving the Kubernetes scheduler per-pod control over specific GPU instances rather than opaque integer counts. Demand changes trigger placement adjustments automatically rather than a ticket to re-tune a ConfigMap. Where VRAM is the binding constraint, method selection drives outcomes. Time-slicing across any NVIDIA GPU handles workloads whose combined memory fits within device VRAM. MIG support extends to A100, A30, H100, H200, and Blackwell-generation GPUs for workloads that need hardware-isolated memory budgets and deterministic per-partition throughput. MPS, providing concurrent CUDA execution with configurable SM partitioning, is available on GCP GKE see the comparison table for roadmap status on additional cloud providers . One A100, partitioned into seven MIG instances with four time-sliced replicas each, presents 28 logical GPU slots to the Kubernetes scheduler. This is a ceiling, not a starting point. Throughput per slot decreases at high density, so test under representative load before using this configuration for latency-sensitive inference. The automation layer keeps a validated configuration stable when new models deploy, instead of letting it decay until the queue returns. ALLEN Digital: 71% cost reduction by sharing seven models Dedicated GPU instances for intermittent workloads are a billing problem before they become a capacity problem. The ALLEN Digital case study https://cast.ai/case-studies/allen-digital/ shows what that looks like at production scale. Seven ML models ran on SageMaker: three open-source BGE-M3, LlamaGuard, Multilingual E5 Large and four custom-built. Each held its own dedicated GPU instance continuously. Load was bursty student queries; instances billed around the clock regardless. SageMaker offered no path to consolidate those models onto shared capacity, so the team evaluated alternatives. On EKS, Cast AI’s inference optimization product, Kimchi Inference, managed GPU time-slicing, node bin-packing, and a 50/50 on-demand/Spot instance split. Together, these changes cut ALLEN Digital’s costs by 71% compared to SageMaker while latency held. Of that 71% total reduction, time-slicing contributed approximately 20 percentage points. Consolidating models onto shared GPU instances drove another 30 to 40 percentage points. Spot adoption and CPU/memory rightsizing closed the remainder. BGE-M3 hit compatibility issues with time-slicing at the start. Resolution took two days from the Cast AI team. After the fix, latency came in below the prior SageMaker baseline. Karthik Bhat, DevOps Engineer 2 at ALLEN Digital, put it directly: “If your models are underutilized, or if you’re trying to achieve higher utilization and fully leverage GPU capacity while reducing costs, I think Kimchi Inference is a great solution.” That 71% is a SageMaker-to-EKS number. It reflects sharing, Spot adoption, and rightsizing working together, not any single lever. What transfers across environments is the structural pattern: dedicated allocation on intermittent inference creates inefficiency that GPU sharing can recover, when the workload memory profile allows it. Conclusion Queued workloads beside idle-looking GPUs are almost never a hardware shortage. They reflect a scheduling, allocation, memory, or application-layer configuration problem. The diagnostic sequence in this post gives you the commands to identify which one applies. GPU sharing addresses the allocation case; VRAM constraints call for memory partitioning via MIG or MPS with SM partitioning. Placement failures resolve through scheduling configuration, not additional hardware. The application-layer bottleneck requires CPU resources, batching adjustments, or data pipeline work, not more GPUs. In practice, teams encounter the allocation constraint first and the application-layer bottleneck last, but skipping the middle two is how you end up buying GPUs that won’t resolve the wait. Frequently Asked Questions Why can GPU workloads queue when utilization is low? Kubernetes allocates GPUs exclusively by default. When a pod claims a device, no other pod can use it regardless of how much compute or memory the occupying workload actually consumes. A GPU reporting 5% compute activity is still fully allocated from the scheduler’s perspective. Queued workloads cannot land on it until the occupying pod releases the device or the cluster is configured to use GPU sharing. Low utilization and available capacity are not the same thing. What is the difference between GPU allocation and GPU utilization? GPU allocation is a Kubernetes scheduling state: a device is either available for new pods or it is not. GPU utilization measures compute activity percent of SMs active or memory occupancy on a device that is already running workloads. A device can be fully allocated with zero availability for new workloads while reporting low compute utilization, because the occupying workload is idle between requests but continues to hold the device and its VRAM. Can Kubernetes workloads share one GPU? Yes, but it requires changing the default configuration. The NVIDIA GPU Operator supports time-slicing, which allows multiple pods to share a single device through temporal multiplexing. MIG, available on Ampere-generation and newer GPUs, supports hardware-isolated partitions with up to seven instances per physical GPU. MPS enables concurrent CUDA process execution across co-located workloads. None of these are enabled by default. Each method has different isolation properties, hardware requirements, and observability characteristics. How does MIG differ from time-slicing? MIG partitions the physical GPU at the hardware level into up to seven isolated instances, each with separate memory controllers, L2 cache banks, and DRAM buses. One instance’s activity cannot affect another’s throughput or latency. MIG requires Ampere architecture or newer: A100, A30, H100, H200, and Blackwell-generation GPUs B200, GB200 . Time-slicing is software-level temporal multiplexing that works on any NVIDIA GPU, including older generations, but provides no memory or fault isolation between co-scheduled workloads. NVLink is also not supported between MIG instances, which matters for distributed training. Can GPU sharing affect inference latency? Yes. Time-slicing can increase tail latency p99 and above when co-scheduled workloads submit burst requests simultaneously, because the GPU must interleave execution across all processes. MIG eliminates this cross-workload interference through hardware isolation, at the cost of fixed partition sizes and hardware compatibility requirements. MPS reduces context-switching overhead compared to time-slicing but provides limited isolation in its basic configuration; SM and memory partitioning are available from CUDA 11.0 onward for more control. Testing at realistic request rates before enabling any sharing method in production is the only reliable way to quantify the latency impact for a specific workload mix. When should a team add GPUs rather than share existing capacity? After ruling out allocation, memory, and placement constraints through the diagnostic steps described in this post. If every device is memory-saturated near-zero free VRAM at full compute, if workloads cannot tolerate the latency increases that sharing produces, or if request volume genuinely exceeds the throughput ceiling of available hardware at any viable sharing configuration, additional GPUs are warranted. Adding hardware before completing this diagnostic frequently compounds the underlying configuration problem without resolving it.