GPU capacity is the tightest it has been in a while. Amazon just crossed three trillion dollars largely on cloud AI demand and the reporting says even AWS can't add capacity fast enough. When supply is that tight, the price gap between on-demand and spot GPUs gets wide and interesting, and every team running inference is staring at the same question: how much of this can we safely move to spot?
I'm a cloud associate and this is a decision I've had to make with real money attached, so instead of the usual "spot is 70% cheaper, go use it" take, here are the actual rules we use to decide what goes where. Spoiler: the headline discount is the least important number.
Yes, spot GPU instances often run 60-70% below on-demand. That number is real and it's also a trap, because it quietly assumes your workload doesn't care about being interrupted. GPU spot capacity is the first thing reclaimed when demand spikes, which right now is often. So the real comparison isn't "cheap vs expensive," it's "cheap-but-can-vanish vs expensive-but-guaranteed," and the right answer depends entirely on what the workload does when it gets a two-minute eviction notice.
We put every GPU workload into one of three buckets based on a single question, what does an interruption actually cost us.
Bucket 1: interruption is free-ish, go spot. Batch inference, offline embedding jobs, eval runs, anything that can checkpoint and resume. If a job can be killed and restarted with no user impact and minimal lost work, spot is close to a no-brainer. We run these on spot with checkpointing every few minutes so an eviction costs us seconds, not the whole job.
Bucket 2: interruption is survivable if you engineered for it. Real-time inference behind a load balancer, where losing one node degrades but doesn't break service. Spot works here only if you've done the work: spread across multiple instance types and AZs so one capacity pool draining doesn't take you down, keep a small on-demand baseline for the floor, and let spot handle the burst. This is where most of the savings actually live, and also where most of the outages come from when people skip the engineering.
Bucket 3: interruption is unacceptable, stay on-demand (or reserved). Anything user-facing with a hard latency SLA and no graceful degradation, or a single-node workload that can't tolerate a restart. Paying full price here isn't a failure, it's buying reliability you actually need. Trying to force these onto spot is how you end up explaining an incident.
Three line items that don't show up in the "spot is 70% off" pitch and that we learned to budget for:
The most honest thing I can tell you: before you optimize the price of the GPU, check whether the GPU should be running at all. Our biggest inference savings didn't come from spot, they came from scheduling. Eval and dev GPU pools have no business running overnight or on weekends, and utilization on "always-on" inference fleets is usually far below what people assume. We schedule non-production GPU capacity to scale down off-hours the same way we schedule any other non-prod resource (that scheduling is a core part of what ZopNight does for us, but you can do the crude version with a cron job and an autoscaler). A 70% spot discount on a node that shouldn't be on is still 100% waste.
Bucket the workload by interruption cost first, model the blended price including cold starts and your on-demand baseline second, and before any of that, make sure the capacity is even supposed to be running. The spot discount is real, but it's the last lever, not the first.
If you're running inference on spot, which bucket gave you the most trouble? For us it was bucket 2, the "survivable if engineered" tier, where a bad week of evictions taught us to spread across instance types the hard way.