The GPU hourly rate is the number everyone compares. It is also the number that tells you the least about what a training run actually costs.
The sticker price, say $2 to $3.50 an hour for an H100 on a specialized cloud, is the visible tip. The real bill is built from three things almost nobody puts on the comparison spreadsheet: the GPU sitting idle, the data you have to move, and the cost of ever leaving. This post breaks down each one, with 2026 numbers, and what you can actually do about it.
The most expensive line item in most setups is not compute. It is compute you pay for but never use.
The 5 percent problem
A 2026 Cast AI report found average GPU utilization across Kubernetes clusters on major clouds sits around 5 percent. Other analyses are kinder, Anyscale puts sustained production utilization below 50 percent, FinOps studies land at 20 to 30 percent, but the conclusion holds: most of every GPU-hour you pay for produces no useful work.
Why it happens
It is structural, not laziness. Workloads bounce between CPU preprocessing, GPU training, and CPU postprocessing. Python datas on the GPU node starve the accelerator. Teams overprovision to dodge out-of-memory errors and default to the biggest instance "just in case."
Why it hurts more than CPU waste
An idle CPU costs cents per hour. An idle GPU costs dollars per hour. A single AWS p4d.24xlarge left idle over one weekend burns about $1,573 for nothing. A month of overnight and weekend idling typically wastes $3,000 to $8,000 per instance.
What to do
nvidia-smi
that scales down an instance after utilization stays below ~5 percent for 30 minutes is the highest-ROI thing most teams can ship. Commonly cuts 20 to 35 percent off GPU spend.Up data is free everywhere. Moving it out is not, and the asymmetry is deliberate.
The 2026 rates
Verified across multiple pricing surveys: AWS charges about $0.09 per GB outbound, roughly $90 per TB. Google Cloud is higher at $0.12 per GB. Azure sits in the same range. Hetzner includes large free allowances and charges on the order of $1 per TB beyond them. Some object-storage options are zero egress entirely.
Why training amplifies it
The volumes are large and recurring. Datasets, checkpoints, and exported weights all move. On a workload pulling 10 TB out per month, the gap between a hyperscaler and a zero-egress provider is the difference between a four-figure line item and almost nothing.
What to do
The third cost is the one you do not see until you try to escape it. It is the same mechanism as egress, viewed over a longer horizon.
Data gravity is the anchor
Once terabytes of data and checkpoints accumulate in one region, moving them is slow and expensive. The egress fee is not just a per-transfer cost, it is an exit tax that grows with every gigabyte you store. By design, the more your data piles up, the less likely you are to leave.
It is not only data
Proprietary services, custom tooling, and provider-specific orchestration all raise the cost of moving. But for training workloads, raw data gravity is the heaviest anchor.
What to do
The headline GPU rate is the smallest part of the story. A realistic cost model looks more like this:
(GPU rate x hours x utilization gap) + storage + egress + the eventual cost of leaving
Two things follow. First, fixing utilization is usually the fastest win, because you are already paying for that waste today. Second, egress and lock-in are decisions you make once, at the start, that compound for as long as the project runs.
This is why the provider landscape is shifting. Specialized GPU clouds and regional providers increasingly compete on exactly these hidden costs: transparent hourly billing instead of a maze of ancillary fees, and zero egress so your data, and your freedom to move it, stays yours. Orion AI Factory in Europe is one example of that model, and the same logic shows up across a growing set of regional and specialized providers. The common thread is pricing the things that used to hide in the footnotes.
None of this needs exotic tooling. Watch your utilization, keep data close to compute, compress what you move, and know your exit cost before you are locked in. The teams that win on cost are not the ones with the biggest budgets. They are the ones who read past the hourly rate.
References