Kubernetes Spot Instances: How to Cut Compute Costs Without Gambling on Reliability Teams running Spot-heavy Kubernetes configurations save 77% on compute on average, compared to 59% for mixed fleets, according to the 2025 Cast AI Kubernetes Cost Benchmark. AWS gives a 2-minute interruption notice, while GCP and Azure give approximately 30 seconds (best-effort), which changes how pod shutdown is configured. Cast AI predicts Spot interruptions 1 hour ahead on AWS and up to 3 hours ahead on GCP, then proactively rebalances before eviction hits. Key takeaways - Teams running Spot-heavy configurations save 77% on compute on average, compared to 59% for mixed fleets, according to the 2025 Cast AI Kubernetes Cost Benchmark. - AWS gives you a 2-minute interruption notice. GCP and Azure give approximately 30 seconds best-effort , which changes how you configure pod shutdown. - Not every workload belongs on Spot. Stateless services, batch jobs, CI/CD runners, and checkpointed ML training are safe bets. Stateful databases, control-plane components, and payment processors are not. - PodDisruptionBudgets are the most critical safety mechanism for Spot workloads. Without them, a node drain can take down all replicas simultaneously. - Cast AI predicts Spot interruptions 1 hour ahead on AWS and up to 3 hours ahead on GCP, then proactively rebalances before the eviction hits. Why Spot cuts cost so sharply Cloud providers sell unused compute capacity at steep discounts. When demand increases and they need those resources back, they reclaim the instance with a short interruption notice. That is the entire trade-off. You get compute at 60-90% below on-demand rates in exchange for accepting occasional, predictable-ish eviction. For workloads designed to survive node loss, this is a reasonable deal. The 2025 Cast AI Kubernetes Cost Benchmark shows clusters running Spot-heavy configurations save 77% on compute. Even mixed fleets partial Spot coverage average 59% savings. On a $50k/month EC2 bill, that difference pays for engineering time. | Dimension | Spot | On-Demand | |---|---|---| | Cost | 60-90% below on-demand | Baseline | | Availability | None — provider can reclaim | Guaranteed | | Interruption notice | AWS: 2 min. GCP/Azure: ~30 sec best-effort | None | | Best workload types | Stateless, batch, CI/CD, ML checkpointed | Stateful DBs, control plane, payments | | Risk level | Medium manageable | Low | | Pricing stability | Variable; GCP most stable, Azure EU most volatile | Fixed | Savings vary by cloud and region Not all Spot markets behave the same. AWS reprices approximately 197 times per month, so your savings vary constantly. GCP reprices far less often roughly 0.35 times per month , which makes GCP Spot pricing more predictable. Azure is the most volatile in EU regions: germanywestcentral saw a 150% price increase between 2022 and 2023, australiaeast climbed 131%, and westeurope rose 130%. Pool size matters too. AWS us-east-1 and us-west-2 have the largest available capacity, which means lower interruption rates. For APAC workloads, ap-southeast-1 Singapore and ap-northeast-1 Tokyo offer far better availability than newer regions. Avoid af-south-1 and ap-southeast-4 for anything latency-sensitive; AWS Spot Advisor shows both in the highest interruption frequency buckets. On GCP, us-central1 and us-east1 provide the most stable Spot availability globally. For more on EKS-specific cost levers, see the EKS cost optimization guide https://cast.ai/blog/eks-cost-optimization/ . For GKE, see the GKE cost optimization guide https://cast.ai/blog/gke-cost-optimization/ . One important GCP distinction: GCP offers two discount VM types. Preemptible VMs use the older API and carry a hard 24-hour maximum runtime limit alongside the 30-second notice. GCP Spot VMs are the current recommendation: no 24-hour limit and the same 30-second best-effort notice. For any new GKE workloads, use Spot VMs, not Preemptible. Which workloads suit Spot Safe to run on Spot Stateless web services recover quickly from interruption, especially when you run at least two replicas and configure readiness probes correctly. Batch jobs and data processing pipelines are natural fits: a terminated job restarts from the last checkpoint with no lasting damage. CI/CD runners are arguably the best Spot use case because each job is ephemeral by design. ML training works on Spot when jobs checkpoint to durable storage every few minutes, so a preemption costs you minutes, not hours. Not safe for Spot Stateful databases, including Postgres, MySQL, and Kafka, should stay on on-demand nodes. A forced eviction mid-write can corrupt data or cause replication lag that takes hours to recover. Your Kubernetes control plane etcd, kube-apiserver must never run on Spot. Payment processors and anything with strict SLA commitments belong on guaranteed capacity. Conditional Some workloads work on Spot with the right architecture. Redis works if it is used purely as a cache not as a primary store and you tolerate cold-start latency on reconnection. Kafka consumers work on Spot if partition rebalancing is fast enough for your consumer group SLO. Evaluate your rebalancing time under load before committing. Interruption handling and fallback Spot reliability comes from how well your stack handles eviction, not from avoiding it. Two-minute AWS notice sounds generous until you realize your pod shutdown sequence needs to finish within that window. On GCP and Azure with 30-second best-effort notice, the math gets tight. Here is how to build each layer of the stack correctly. Node-level: AWS Node Termination Handler If you use Karpenter, you do not need NTH. Karpenter watches for Spot interruption notices via EventBridge and IMDS, cordons the affected node, and begins provisioning a replacement — all natively, without a separate DaemonSet. NTH is for clusters using Cluster Autoscaler or self-managed node groups without Karpenter. For everyone else, the AWS Node Termination Handler NTH picks up the Spot interruption notice, cordons the node, and drains pods gracefully before the instance disappears. The right NTH mode depends on how your nodes are managed: Cluster Autoscaler + self-managed node groups: Use NTH IMDS polling mode. It queries the instance metadata service directly, requires no additional AWS infrastructure, and is the simpler setup. Cluster Autoscaler + EKS managed node groups: Use NTH Queue Processor mode. This requires both EventBridge and SQS: EventBridge captures the interruption event and routes it to an SQS queue, which NTH then polls. If you run managed node groups and skip the SQS queue, NTH will not receive interruption events for those nodes. The Helm chart for Queue Processor mode requires --set enableSqsTerminationDraining=true along with your queue URL. Pod-level: grace periods and preStop hooks The terminationGracePeriodSeconds field belongs at the pod spec level, not inside the container spec. This is a common YAML mistake that causes Kubernetes to silently ignore the value and use the default 30 seconds instead. The right grace period depends on which cloud you are on. Each cloud has a different interruption notice window, and your preStop hook plus application shutdown must fit within that ceiling. AWS Spot 2-minute notice AWS gives you 2 minutes from interruption notice to instance termination. A 90-second grace period leaves a 5-second preStop hook and 85 seconds for in-flight request draining: spec: terminationGracePeriodSeconds: 90 Within 2-minute AWS Spot interruption notice containers: - name: app lifecycle: preStop: exec: command: "/bin/sh", "-c", "sleep 5" GCP Spot VMs 30s best-effort notice GCP Spot VMs provide a 30-second best-effort eviction notice. Keep the total grace period inside that window. A 2-second preStop hook and 23 seconds of application shutdown fits comfortably within the 25-second ceiling: spec: terminationGracePeriodSeconds: 25 GCP Spot VMs: conservative ceiling within 30s notice containers: - name: app lifecycle: preStop: exec: command: "/bin/sh", "-c", "sleep 2" Note: GKE’s managed node shutdown process can give up to 120 seconds in some scenarios, but plan conservatively for the 30-second guaranteed window. GCP Preemptible VMs and Azure Spot shorter effective window GCP Preemptible VMs and Azure Spot both warrant extra care. Azure Spot gives 30 seconds notice, matching GCP Spot VMs. However, GCP Preemptible VMs have a hard shutdown at approximately 15 seconds from eviction signal, making the effective window shorter than the stated 30 seconds: spec: terminationGracePeriodSeconds: 25 GCP Preemptible effective max ~15s ; Azure Spot 30s notice containers: - name: app lifecycle: preStop: exec: command: "/bin/sh", "-c", "sleep 2" Note: GCP Preemptible VMs have a hard shutdown at approximately 15 seconds from eviction signal. For Preemptible, shorten preStop to 2 seconds and minimize shutdown logic. For new GKE workloads, use Spot VMs instead to avoid this constraint and remove the 24-hour maximum runtime limit. PodDisruptionBudgets: the safety mechanism most teams skip A PodDisruptionBudget PDB tells Kubernetes the minimum number of replicas that must stay available during voluntary disruptions, which includes node drains triggered by NTH. Without a PDB, a drain operation can evict every pod in a Deployment simultaneously, taking your service down to zero replicas. PDBs have two configuration options: minAvailable specifies the minimum number of pods that must stay running, and maxUnavailable specifies the maximum number that can be down at once. For most production services, minAvailable: 1 is the right starting point: apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: my-app-pdb spec: minAvailable: 1 selector: matchLabels: app: my-app One critical warning: do not set maxUnavailable: 0 . This configuration blocks drain operations entirely. If NTH attempts to drain a node before eviction and the PDB blocks all disruptions, the pods will not move before the instance terminates. Use minAvailable: 1 instead, which allows drain to proceed as long as at least one replica stays up. NTH respects PDBs during the drain phase. When NTH cordons a node and issues eviction requests, Kubernetes checks the PDB before evicting each pod. If evicting the pod would violate the budget, Kubernetes waits until a replacement pod is available elsewhere. This coordination between NTH and PDBs is what makes graceful Spot drains work reliably at scale. Fallback to on-demand When Spot capacity disappears, your workload needs somewhere to land. Node affinity with a preferredDuringSchedulingIgnoredDuringExecution rule tells the scheduler to try Spot first and fall back to on-demand if no Spot node is available: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 preference: matchExpressions: - key: karpenter.sh/capacity-type operator: In values: - spot The label karpenter.sh/capacity-type is set by Karpenter on nodes it provisions. If you use Cluster Autoscaler instead, use the node group label defined in your autoscaler node group tags. The label name differs depending on your tooling and how you tag node groups. Spot with Karpenter Karpenter changes the Spot game significantly. Rather than pre-defining node groups with specific instance types, Karpenter selects from a broad pool of compatible instances in real time. This means it can automatically pick Spot instance types with lower interruption frequency when higher-frequency types become volatile. For a deeper walkthrough of Karpenter’s Spot mechanics, see the Karpenter Spot instances guide https://cast.ai/blog/karpenter-spot-instances/ . The key Karpenter configuration for Spot is the NodePool’s capacity type and instance family diversity. Defining at least 15 compatible instance types in your NodePool gives Karpenter enough flexibility to find available Spot capacity across multiple pools. A minimal NodePool targeting Spot with on-demand fallback looks like this: apiVersion: karpenter.sh/v1beta1 kind: NodePool metadata: name: default spec: template: spec: requirements: - key: karpenter.sh/capacity-type operator: In values: "spot", "on-demand" - key: node.kubernetes.io/instance-type operator: In values: - m5.large - m5a.large - m5n.large - m4.large - m6i.large - m6a.large - m6in.large - c5.xlarge - c5a.xlarge - c5n.xlarge - c6i.xlarge - c6a.xlarge - c6in.xlarge - r5.large - r6i.large SpotToSpotConsolidation Karpenter v0.34.0 released March 2024 introduced SpotToSpotConsolidation, which allows Karpenter to replace a running Spot node with a cheaper Spot option when one becomes available. This feature is disabled by default and requires two conditions: first, your NodePool must include 15 or more compatible instance types; second, you must enable the feature gate explicitly: helm upgrade karpenter oci://public.ecr.aws/karpenter/karpenter \ --version