# Maximizing Apache Spark availability: Mitigating compute stockouts with flexible VMs and other best practices

> Source: <https://cloud.google.com/blog/products/data-analytics/maximize-apache-spark-availability-with-flexible-vms/>
> Published: 2026-09-21 16:00:00+00:00

The surge in AI development has created unprecedented demand for compute capacity around the globe. This can have negative implications for data processing and pipelines with Apache Spark. Whether you are managing your own Spark infrastructure or using a managed service, you can face availability constraints. However, a significant advantage of using Google’s [Managed Service for Apache Spark](https://cloud.google.com/products/managed-service-for-apache-spark) is the availability of [flexible VMs,](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/flexible-vms) which provide a targeted mechanism to adopt a dynamic, resource-agnostic philosophy and ensure your pipelines remain operational, even during regional or zonal capacity stockouts.

Capacity stockouts occur when demand for a specific machine family (such as N2 or N2D) exceeds available capacity in a target zone or region. For time-sensitive analytics pipelines, rigid single-VM requirements transform standard provisioning into a single point of failure which can result in cluster creation delays, failed executions, and potentially compromised business SLAs.

Flexible VMs fundamentally overhaul how a Managed Spark cluster requests compute resources. Rather than binding a cluster to a rigid instance type, flexible VMs allow teams to establish an ordered list of acceptable machine families for master, primary worker, and secondary worker nodes.

**Multi-family blending:** Mix nodes across diverse machine types and generations, combining Gen2 families (e.g., N2, N2D) with Gen4 families (e.g., N4, C4) in a single configuration.

**Mixed storage support:** Broaden available capacity pools by allowing storage options to dynamically adapt to the underlying host family's supported disk types.

**Comprehensive cluster coverage:** Apply flexible rules to primary workers, secondary (preemptible/spot) workers, and master nodes to guarantee cluster provisioning end-to-end.

A successful flexible VM implementation relies on intentional ranking. By defining a clear hierarchy of options, Managed Spark clusters automatically attempt provisioning, systematically mitigating stockout risks without requiring manual intervention. To improve the availability of suitable VMs, we recommend specifying at least two machine families in the highest priority (Rank 0) flexible VM list.

As an example, for production pipelines standardizing on **n2d-standard-16** shapes, the following tiering strategy provides robust resilience against capacity constraints:

| **Rank** | **Machine family examples** | **Storage recommendation** | 
|---|---|---|
| **Rank 0 (Primary)** | n2d-standard-16, n2-standard-16 | Standard Local SSD or PD | 
| **Rank 1** | n4-standard-16, n4d-standard-16 | Hyperdisk Balanced | 
| **Rank 2** | c4-standard-16, c3-standard-22 | Hyperdisk Balanced | 
| **Rank 3**  | e2-standard-16 | Standard PD | 

For pipelines standardizing on legacy **n1-standard-16** shapes, the following tiering strategy helps transition workloads toward newer, more available architectures while preserving operational stability:

| **Rank** | **Machine family examples** | **Storage recommendation** | 
|---|---|---|
| **Rank 0 (Primary)** | n1-standard-16 n2-standard-16 | Standard Local SSD or PD | 
| **Rank 1** | n2d-standard-16 | Standard Local SSD or PD | 
| **Rank 2** | n4-standard-16 n4d-standard-16 | Hyperdisk Balanced | 
| **Rank 3** | e2-standard-16 | Standard PD | 

Unlocking maximum availability with flexible VMs often requires adopting modern storage architectures like [Hyperdisk Balanced](https://docs.cloud.google.com/compute/docs/disks/hd-types/hyperdisk-balanced). Newer instance families (including N4 and C4) rely on Hyperdisk to deliver predictable performance across variable VM sizes. Starting with default IOPS and throughput settings typically provides a reliable baseline for the majority of distributed Spark jobs.

While flexible VMs dramatically improve cluster provisioning success, aligning them with enterprise requirements involves evaluating several architectural and financial factors:

It is no longer enough to have one specific machine (e.g., N2) quota. You need to ensure you have sufficient compute and disk quotas allocated for all specific machine types and disks (including Hyperdisk) defined in their flexible VM lists.

Traditional, resource-based CUDs are tied to specific machine families, which limits flexibility. Adopt [Compute flexible Committed Use Discounts (CUDs)](https://docs.cloud.google.com/compute/docs/instances/committed-use-discounts-overview#spend_based) to apply savings across multiple VM families and regions.

Performance can vary between machine generations, as well as between Local SSD and Hyperdisk. While the Managed Spark team maintains [internal benchmarks for these comparisons](https://docs.cloud.google.com/compute/docs/machine-resource), actual outcomes are workload-dependent. Testing your specific Spark jobs across these families is essential for understanding SLA impacts.

In addition to implementing flexible VMs, there are several other key architectural and scheduling strategies to improve resource availability and workload stability:

**AutoZone:** Implement [AutoZone](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/auto-zone) routing to allow Managed Spark to automatically select the zone best suited to execute the job based on current capacity.

**Smaller machine shapes:** Avoid high in demand, large-core shapes. Design workloads and YARN containers to utilize [smaller machine shapes](https://docs.cloud.google.com/compute/docs/general-purpose-machines) (such as 4, 8, or 16 cores). These smaller shapes are much easier to fulfill from the available GCE on-demand pool.

**Autoscaling:** Deploy cluster [autoscaling](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/autoscaling) with reasonable `maxInstances` to manage capacity effectively for bursty or unpredictable workloads without relying on rigid, massive upfront provisioning.

**Partial cluster creation****:** Configure a minimum acceptable number of primary workers. This allows clusters to spin up under resource constraints and begin executing, while autoscaling can dynamically add remaining workers as resources become available.

**Establish regional fallbacks:** Some regions, such as `us-central1,` can experience  high demand. Setting up fallbacks to other regions reduces capacity stockout risks.

Managing your own Apache Spark infrastructure can be complex, especially when capacity stockouts disrupt your data processing. Utilizing a managed service like Managed Service for Apache Spark provides unique advantages — including built-in platform resilience and access to flexible VMs. By adopting a prioritized fallback strategy with flexible VMs, you can protect your workloads from regional hardware shortages and keep your critical pipelines running.

Ready to improve your Spark workload resilience? Start configuring [flexible VMs](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/flexible-vms) for your Managed Spark clusters today.