On January 4, 2026, AWS raised H200 GPU instance prices by 15%. This was the first GPU price increase in roughly two decades. Yet across more than 23,000 Kubernetes clusters in our dataset, the average GPU utilization sits at just 5%. So before that price increase even landed, most teams were already paying for twenty times more GPU capacity than they actually used. The pricing environment just got harder. The utilization problem has not changed.
Key takeaways #
- AWS raised H200 (p5e, p5en) on-demand prices by 15% on January 4, 2026, marking the first GPU price increase in roughly two decades.
- H100 on-demand costs range from $12.29/GPU/hr on AWS (p5.48xlarge) to $13/GPU/hr on Azure. The same GPU type costs up to 6% more depending on which cloud you use.
- Spot and preemptible instances cut costs by 60–91%, but H200 spot availability remains limited across all major clouds.
- Average GPU utilization across Kubernetes clusters is 5%. At that rate, effective GPU cost runs 20x the nominal hourly rate.
- GPU sharing, idle detection, and multi-cloud sourcing reduce effective GPU cost without requiring a change in instance type.
What drives GPU pricing #
GPU cloud costs don’t follow a simple supply-and-demand curve. Instead, several compounding forces push prices up and keep them volatile. Understanding these forces helps you predict where pricing heads next and plan procurement accordingly.
Supply constraints at the chip level
NVIDIA has orders for 2 million H200 chips but only 700,000 in available supply. TSMC is currently prioritizing Blackwell production, which squeezes H200 output further. Additionally, HBM3E memory costs have risen 20%. These constraints feed directly into cloud provider pricing. When chip supply lags demand, cloud providers allocate scarce capacity to the highest bidders, and spot markets thin out fast.
AI demand outpacing infrastructure build-out
Training and inference workloads are scaling faster than data center capacity. As a result, demand consistently exceeds supply at the top end of the GPU stack. This dynamic keeps H100 and H200 on-demand prices elevated even as A100 costs have stabilized or declined in some markets. More teams running more models means the pressure on GPU availability is not easing soon.
Architecture transitions creating uneven pricing
The shift from H100 to Blackwell is creating a two-speed market. H100 faces emerging oversupply risk as H200 and B100 ramp up. However, Blackwell availability remains limited in 2026. So H200 commands a premium while H100 spot prices have dropped sharply: in certain AWS regions, H100 Spot prices fell by as much as 88% from January 2024 to September 2025, reflecting the growing supply of H100 inventory as data centers scaled capacity. H200 spot inventory, by contrast, stays thin. Teams planning around Blackwell will find the on-demand market tight throughout 2026.
Commitment structures and regional variation
1–3 year reserved contracts dominate high-end GPU supply. On-demand access at scale is difficult to secure during demand spikes. Regional factors compound the issue further. Spot availability, data center capacity, and cross-AZ egress charges all affect the true cost of a GPU workload. A spot H100 in us-east-1 costs differently than the same instance in eu-west-1. Teams running multi-region architectures need to account for these differences in their cost models.
2026 GPU pricing across clouds: H100, H200, spot vs on-demand #
The following table shows on-demand GPU pricing as of mid-2026. Spot savings reflect approximate market discounts and vary by region and availability. For the full dataset and historical trends, see the Cast AI GPU Trends and Cost Report.
| Cloud | Instance | GPUs | GPU Type | On-Demand/hr | Per GPU/hr | Spot Savings |
|---|---|---|---|---|---|---|
| AWS | p4d.24xlarge | 8 | A100 40GB | ~$32.77 | ~$4.10 | Up to 80% |
| AWS | p5.48xlarge | 8 | H100 80GB SXM5 | ~$98.32 | ~$12.29 | Up to 70% |
| AWS | p5e.48xlarge | 8 | H200 141GB HBM3e | $39.80 | ~$4.98 | Limited |
| AWS | p5en.48xlarge | 8 | H200 141GB HBM3e | $41.61 | ~$5.20 | Limited |
| GCP | a2-highgpu | 1–8 | A100 | Varies | $2.70–$3.70 | Up to 91% |
| GCP | a3-highgpu-8g | 8 | H100 80GB | ~$80–$90 | $9–$11.50 | Up to 70% |
| Azure | NC A100 v4 | 1–4 | A100 80GB | Varies | $3.50–$4.20 | Up to 80% |
| Azure | ND H100 v5 | 8 | H100 80GB | ~$98 | $11–$13 | Up to 60% |
Note: AWS H200 pricing shown above reflects Capacity Block pricing (1-day or 14-day reservations). On-demand H200 availability on AWS is limited. Capacity Blocks require a commitment upfront, unlike on-demand pay-per-use billing. The H200 per-GPU-hour rate appears lower than H100 because these are Capacity Block commitment rates, not on-demand pricing. Direct on-demand H200 pricing is not generally available on AWS.
AWS GPU pricing
AWS offers the widest GPU instance range and the lowest H100 on-demand price among major clouds. The p5.48xlarge runs 8x H100 80GB SXM5 at roughly $12.29/GPU/hr on-demand, making it the most widely available entry point for H100 workloads on AWS. Following the January 2026 increase, the p5e.48xlarge (H200) Capacity Block pricing stands at $39.80/hr for 8 GPUs, or about $4.98/GPU/hr – note this is a committed reservation rate, not standard on-demand pricing. Spot availability for H200 remains limited, which reduces cost-optimization options for teams relying on preemptible capacity. For Kubernetes GPU optimization at scale, AWS still delivers the best combination of on-demand price and spot market depth for H100.
GCP GPU pricing
The GCP H100 (a3-highgpu-8g) runs $80–$90/hr for 8 GPUs ($9–$11.50/GPU/hr on-demand), which is somewhat lower than AWS per-GPU pricing of $12.29/GPU/hr. GCP preemptible instances offer up to 91% savings on A100, making GCP competitive for fault-tolerant training workloads on older GPU hardware. For inference workloads that tolerate interruption, GCP’s A100 preemptible pricing at $2.70–$3.70/GPU/hr represents the lowest effective cost in the market. Azure ND H100 v5 remains the most expensive option at $11–$13/GPU/hr on-demand.
Azure GPU pricing
Azure’s ND H100 v5 is the most expensive H100 option at $11–$13/GPU/hr on-demand. A full 8-GPU node runs roughly $98/hr. Spot savings reach up to 60%, which is the lowest of the three major clouds for H100. For teams already running on Azure, the cost premium may fit within existing enterprise agreements. For teams choosing a cloud from scratch based on H100 unit economics, AWS delivers a significant discount.
Why utilization beats unit price #
The pricing table above matters. But it captures only half the cost picture. The more important number is utilization.
According to our 2026 State of Kubernetes Optimization Report, the average GPU utilization across 23,000+ Kubernetes clusters is 5%. AKS clusters average 2%. EKS clusters average 5%. GKE clusters average 6%. These numbers hold across large production environments, not just experimental clusters.
The real cost math
At 5% utilization, your effective GPU cost is 20x the nominal rate. The formula is straightforward:
Effective GPU cost = hourly rate / actual utilization rate
At 5% utilization: $12.29 / 0.05 = $245.80 per GPU-hour of compute actually delivered
Each idle H100 costs roughly $8,850/month at AWS on-demand rates. Scale that to a team of 20 data scientists running with idle GPU habits, and wasted H100 spend reaches $177,000 per month or more. That figure assumes nothing about bad instance choices. It’s pure idle waste on correctly sized instances.
Consequently, a 10-percentage-point improvement in GPU utilization reduces effective cost more than switching from Azure’s most expensive H100 tier to AWS’s cheapest one. Unit price comparisons matter for procurement. Utilization is the primary lever for ongoing cost control.
Why utilization stays low
GPU nodes typically stay allocated long after a job finishes. Notebooks and interactive sessions hold GPUs throughout the workday, even when idle. Multi-GPU jobs often under-utilize individual GPUs due to poor bin-packing. Additionally, teams frequently over-provision to avoid job failure, which pads utilization buffers that never get used. These are not exotic problems. They appear in 95% of the clusters we analyze, across every major cloud.
How to lower effective GPU cost #
There are five proven tactics for reducing effective GPU cost. Some apply at the scheduling layer. Others require changes to how you source compute. Together, they can cut GPU spend by 40–70% without touching your model or training code.
1. GPU sharing via MIG and time-slicing
NVIDIA Multi-Instance GPU (MIG) partitions an H100 into up to 7 isolated instances, each with dedicated memory and compute. Time-slicing gives multiple workloads sequential access to a single GPU. Both approaches increase utilization per purchased GPU. For teams running many small inference workloads, GPU sharing in Kubernetes can multiply effective throughput without adding nodes. The tradeoff is slightly higher scheduling complexity and reduced isolation between tenants.
2. Idle detection and scale-to-zero
DCGM (Data Center GPU Manager) metrics expose real-time GPU utilization at the process level. By integrating these metrics into your autoscaler, you can detect idle GPUs and drain nodes automatically. Teams that implement idle detection and scale-to-zero typically recover 30–50% of their GPU spend within the first billing cycle. This is the fastest-payback optimization in the stack.
3. Spot and preemptible instance automation
Spot and preemptible instances offer 60–91% savings over on-demand for fault-tolerant workloads. In certain AWS regions, H100 Spot now runs approximately $1.95–$2.50/GPU/hr, down as much as 88% from early 2024 peaks in those regions, reflecting growing H100 inventory as data centers scaled capacity. The key requirement is workload checkpointing: training jobs need to resume from the last checkpoint when a spot instance is reclaimed. For inference, stateless containerized services handle preemption well. For guidance on managing GPU shortage and availability risk, automation that spans multiple instance types and fallback options is essential.
4. Multi-cloud GPU sourcing
No single cloud has consistent GPU availability across all regions and instance types. Multi-cloud sourcing lets you pull from whichever provider has capacity at the best price at the time of scheduling. This approach is particularly effective for training jobs that aren’t latency-sensitive. Multi-cloud GPU on Kubernetes requires a workload orchestration layer that can route jobs across clouds transparently. When availability tightens on one cloud, jobs shift to another without manual intervention.
5. Bin-packing before provisioning
Before adding a new GPU node, consolidate existing workloads onto fewer nodes. Bin-packing identifies partially utilized nodes and migrates workloads to fill them before requesting new capacity. This step alone prevents a significant fraction of unnecessary node provisioning. Teams often skip bin-packing because it requires scheduler-level visibility into GPU utilization. Automating it consistently is where most of the gains come from.
Conclusion #
GPU cloud pricing in 2026 is more expensive and more volatile than it was 12 months ago. AWS raised H200 prices for the first time in two decades. Azure H100 on-demand runs up to $13/GPU/hr. Supply constraints on H200 and Blackwell chips keep top-tier GPU pricing elevated for the foreseeable future.
However, the unit price conversation misses the bigger opportunity. At 5% average GPU utilization, the effective cost of cloud GPUs is already 20x higher than the hourly rate. Fixing utilization delivers more cost reduction than any instance-type optimization.
Cast AI automates the full stack of GPU cost optimization: idle detection, scale-to-zero, spot automation, multi-cloud sourcing, and bin-packing. Organizations using Cast AI’s GPU optimization reduce effective compute costs by 40–70% through a combination of idle detection, GPU sharing, and automated scale-down.
Building idle detection and GPU cost attribution manually – monitoring dashboards, custom alerting controllers, DCGM integration, chargeback rollouts – typically takes 4–8 weeks of engineering time. Cast AI delivers GPU utilization monitoring, cost attribution by team, and automated scale-to-zero within hours of connecting your cluster. For a cluster burning $177K/month in idle H100 spend, the payback period is typically days, not quarters.
Frequently Asked Questions #
How much does a GPU cost in the cloud? Cloud GPU pricing in 2026 ranges from approximately $2.70/GPU/hr for an A100 on GCP preemptible to $13/GPU/hr for an H100 on Azure on-demand. AWS H100 on-demand (p5.48xlarge) costs roughly $12.29/GPU/hr. H200 on AWS (p5e) runs about $4.98/GPU/hr via Capacity Block reservations after the January 2026 price increase. Spot and preemptible instances reduce these rates by 60–91% for fault-tolerant workloads. Actual cost depends on the cloud provider, region, GPU generation, and whether you use on-demand or preemptible capacity.
H100 vs H200 pricing: which is better value? For most teams in 2026, H100 and H200 pricing requires careful comparison because the numbers reflect different commitment structures. AWS H100 (p5.48xlarge) costs roughly $12.29/GPU/hr on-demand, while AWS H200 (p5e.48xlarge) costs approximately $4.98/GPU/hr via Capacity Block reservations. Note that H200 pricing here reflects AWS Capacity Block rates (pre-committed reservations), while H100 rates above are on-demand. On-demand H200 pricing is higher and not widely available. H200 delivers higher memory bandwidth (141GB HBM3e) and better performance for large model inference. The value comparison depends on workload type: for memory-bound inference jobs with models exceeding 80GB, H200 may justify the commitment. For training jobs that fit within H100 memory, H100 at lower cost and better spot availability is typically the better choice. Additionally, H200 spot markets remain thin, limiting preemptible savings that make H100 spot very attractive at $1.95–$2.50/GPU/hr.
Which cloud has the cheapest GPUs? For H100 on-demand, GCP’s a3-highgpu-8g runs $9–$11.50/GPU/hr, which is lower than AWS’s $12.29/GPU/hr (p5.48xlarge). Azure is consistently the most expensive option for H100, at $11–$13/GPU/hr on-demand. GCP also offers the deepest spot discounts on A100, with preemptible savings up to 91% and effective rates as low as $2.70/GPU/hr — the lowest effective per-GPU cost in the market for fault-tolerant workloads. For H100 on-demand unit economics, GCP offers lower per-GPU pricing than AWS, though AWS provides better spot market depth and availability for H100. The best choice depends on your GPU generation, tolerance for spot interruptions, and existing cloud commitments.
Why is my effective GPU cost higher than the hourly rate? Because you’re paying for GPU capacity that isn’t doing useful work. The average GPU utilization across Kubernetes clusters is 5%, according to Cast AI’s analysis of 23,000+ clusters. At that utilization rate, effective GPU cost is 20x the nominal hourly rate. GPU nodes stay allocated after jobs finish, notebooks hold GPUs through idle periods, and over-provisioning pads utilization buffers that never get used. The effective GPU cost formula is: hourly rate divided by actual utilization rate. At $12.29/GPU/hr (AWS p5.48xlarge H100 on-demand) and 5% average utilization, the effective cost per utilized GPU-hour reaches $245.80/GPU/hr. That is 20 times the nominal rate, because 95% of reserved capacity sits idle. Fixing idle GPU waste through detection, scale-to-zero, and GPU sharing reduces effective cost faster than switching cloud providers.