# What’s new in Managed Service for Apache Spark 4.0

> Source: <https://cloud.google.com/blog/products/data-analytics/whats-new-in-managed-service-for-apache-spark-4-0/>
> Published: 2026-11-20 17:00:00+00:00

**Key takeaway:** Managed Service for Apache Spark 4.0 standardizes serverless and managed clusters on Apache Spark 4.2, Java 25, Python 3.13 with Pixi, and Apache Iceberg 1.12, and attaches every workload to the Lakehouse runtime catalog instead of a legacy Hive Metastore.

As organizations move from reactive analytics to proactive systems of action powered by autonomous AI agents, data platforms face a new set of demands. Moving from human scale to agent scale changes both how data pipelines run and how they are built: Teams and AI agents now use Model Context Protocol (MCP) tools and agentic troubleshooting to build, submit, and diagnose Apache Spark pipelines while preparing multimodal datasets, performing feature engineering, and acting on live business signals around the clock. Over the past year, customer use of [Managed Service for Apache Spark](https://cloud.google.com/products/managed-service-for-apache-spark) in serverless deployments has nearly doubled, with millions of jobs running each month as enterprises build on Google’s [Agentic Data Cloud](https://cloud.google.com/data-cloud).

In the Agentic Data Cloud, Apache Spark doesn't operate in isolation. Managed Service for Apache Spark runs on Google Cloud's borderless lakehouse, so Apache Spark and BigQuery can read and write the same tables with high performance. Through Knowledge Catalog, both engines share centralized governance, consistent policies, detailed lineage, and semantic discovery for AI agents.

Today, we are announcing the general availability of Managed Service for Apache Spark 4.0, debuting a next-generation runtime foundation for serverless and managed cluster deployments. Engineered to power both modern enterprise data pipelines and fast-growing agentic workloads, Managed Service for Apache Spark 4.0 introduces:

**A new 4.0 runtime with Apache Spark 4.2:** Standardized on Apache Spark 4.2, Java 25, Python 3.13 with Pixi, and Apache Iceberg 1.12, it is now available for both serverless and managed clusters.

**A redesigned, code-first Google Cloud console and agentic troubleshooting:** Onboard quickly with a new Overview page featuring a samples gallery, Lakehouse public datasets, and one-click VS Code authoring in Cloud Shell, manage all serverless and cluster workloads from a unified Jobs view, and use an AI agent to automatically diagnose job failures.

**Third-party IDE integration with the Data Agent Kit:** Configure MCP servers, skills, and API access to build and run Apache Spark pipelines directly from VS Code, Claude Code, and Antigravity.

**Faster startups and Lightning Engine acceleration:** Run on default Gen4 infrastructure with faster startups at no additional cost, and accelerate workloads with Lightning Engine for up to 3.25x the price-performance over the leading high-speed Apache Spark alternative.

**Unified DCU quotas and Lakehouse runtime catalog attachment:** Improve capacity obtainability with serverless infrastructure fungibility and Data Compute Unit (DCU) quotas, while attaching every workload automatically to the [Lakehouse runtime catalog](https://cloud.google.com/products/lakehouse) instead of a legacy Hive Metastore.

This new runtime packages Apache Spark 4.2 (including Spark Declarative Pipelines and Real-Time Mode), Java 25, Scala 2.13, Apache Iceberg 1.12, Delta Lake 4.4, and Python 3.13 with Pixi for faster dependency management.

Managed Spark serverless and cluster workloads now run on the same modern stack whether written by a data engineer or generated by data engineering and data science skills in Gemini, Claude, and other large language models (LLMs). Teams can standardize on one set of component versions and move PySpark, Scala, and Spark SQL workloads between serverless and managed clusters.

Data teams should spend their time writing Spark logic rather than configuring infrastructure, hunting for the right job, or digging through driver and executor logs when something fails. Version 4.0 ships alongside a redesigned, code-first Google Cloud console built to take that operational work off their plate.

The refreshed console streamlines every stage of the Apache Spark lifecycle:

**Frictionless onboarding and one-click authoring:** A redesigned Overview page helps developers get started immediately with an integrated samples gallery, pre-loaded Lakehouse public datasets, guided demos, and one-click authoring in VS Code through Cloud Shell.

**Optimized defaults and reusable runtime profiles:** Out-of-the-box performance defaults simplify job submission and cluster creation. With reusable runtime profiles (in preview) for batch and interactive workloads, platform admins can capture governed hardware, network, and autoscaling baselines once so developers can simply select a profile and run code.

**Unified job, session, and fleet management:** Engineers can submit and monitor batch jobs, interactive serverless sessions, and managed clusters from a single unified listing, backed by streamlined fleet management and redesigned detail views that surface key performance metrics and logs in one place.

**Agentic troubleshooting:** When a job fails or slows down, an AI agent analyzes logs, metrics, and query plans within existing IAM permissions to identify the likely root cause and recommend a fix, helping engineers quickly return a failed job to a healthy state.

Many data engineers and data scientists would rather never leave their editor. With the [Data Agent Kit](https://docs.cloud.google.com/data-cloud-extension), they can build and run Apache Spark workloads from VS Code, Antigravity, Claude Code, and other agentic IDEs. Setup takes a few clicks: pick Apache Spark and the other services you use, and the kit configures Model Context Protocol (MCP) servers, agent skills, and API access for each one. From there, AI agents can generate, tune, and test Apache Spark pipelines on Managed Service for Apache Spark 4.0 without leaving the IDE.

As practitioners and AI agents trigger short-running, ephemeral jobs on demand to prepare data, perform feature engineering, and answer ad-hoc queries, low startup latency and predictable price-performance are essential to avoiding a cost spiral. In Managed Service for Apache Spark 4.0, our performance-optimized serverless experience removes manual infrastructure tuning through three changes:

**Faster startups at no additional cost:** On the serverless Premium tier, runtime and control-plane optimizations in version 4.0 deliver faster startup times than previous runtime versions across interactive sessions and short-running batch workloads at no additional cost.

**Hardware acceleration by default:** New serverless 4.0 workloads run on the Premium tier by default, using Google Cloud's latest-generation [Gen4 VM instances](https://cloud.google.com/compute/docs/general-purpose-machines) and Hyperdisk storage. Teams get our highest-efficiency compute and storage fleet without picking machine types or configuring disks, while aligning infrastructure tiering directly with workload urgency.

**Software acceleration with Lightning Engine:** Our native C++ vectorized execution engine, [Lightning Engine](https://docs.cloud.google.com/managed-spark/docs/guides/lightning-engine), is available in the 4.0 runtime and delivers up to 3.25x the price-performance over the leading high-speed Apache Spark alternative, along with up to 4.9x faster performance compared to running open-source Apache Spark with zero code changes. Built on Apache Gluten and Velox with Google-specific optimizations, Lightning Engine compiles Apache Spark DataFrame, Dataset, and Spark SQL query plans into native SIMD instructions. For lakehouse storage, built-in intelligent caching accelerates file scan times on Apache Iceberg tables by up to 50%. Separately, for workloads querying BigQuery, Lightning Engine's native reader uses Apache Arrow streams to accelerate BigQuery reads by up to 40% compared to default BigQuery reads in Managed Service for Apache Spark. In the 4.0 runtime, Lightning Engine expands support to ARM processors, Ubuntu images, and native vectorized readers for Apache ORC and CSV alongside Apache Parquet, Apache Iceberg, and Delta Lake.

"We were facing not only processing challenges, we were facing management overhead across thousands of servers in our ecosystem, so we migrated to serverless Managed Service for Apache Spark. We moved from managing infrastructure to a no-ops infrastructure. Not only did it help us overcome maintenance and cluster management challenges, it helped us optimize our costs at scale to meet our business SLAs. Apache Spark is no longer just compute for us; it is a strategic business accelerator."**—Saurabh Kumar, Senior Director of Data Engineering, Lowe's.**

When autonomous agents and production pipelines run around the clock, tying a workload to a single machine shape or zone can stall downstream decisions when regional demand spikes. Without dedicated reservations, cloud service providers may not always have available capacity during peak demand. We are introducing infrastructure fungibility for serverless deployments to improve resource availability for all customers:

**Serverless infrastructure fungibility and fallback:** Serverless 4.0 workloads are no longer tied to one VM instance family, disk type, or zone. The platform selects from several latest-generation VM instance families and disk types across zones in the region. If the preferred infrastructure is constrained during a local demand peak, the service automatically chooses the best available infrastructure in the same region instead of failing the job. Once capacity returns, workloads go back to the default Gen4 VM instances and Hyperdisk infrastructure.

**Data Compute Unit (DCU) quotas:** Quotas for serverless deployments are now set in DCUs rather than per machine family. Because the platform selects the VM instance family, admins manage one unified DCU quota instead of separate CPU quotas for each machine type.

A borderless lakehouse breaks down walled gardens by connecting operational, analytical, and cross-cloud data in open formats without moving it. In Managed Service for Apache Spark 4.0, Apache Iceberg ships directly in the runtime image, and every workload automatically has access to the serverless [Lakehouse runtime catalog](https://docs.cloud.google.com/lakehouse/docs/about-lakehouse-catalogs). Teams no longer have to configure a lakehouse metastore with Apache Spark or complete prerequisites such as setting up Google Cloud Storage (GCS) directories beforehand, so they can focus straight on writing business logic. Version 4.0 also eliminates the need for GCS staging buckets and uses end-user credentials by default.

Through the Lakehouse runtime catalog, Managed Service for Apache Spark, BigQuery, and AlloyDB can directly read and write the same Apache Iceberg tables on GCS without ETL pipelines or data copies. Everything registered in the Lakehouse runtime catalog automatically surfaces in [Knowledge Catalog](https://cloud.google.com/products/knowledge-catalog), which enriches metadata and business context so AI agents and Gemini can query and reason over tables curated with Apache Spark with high accuracy, governed access controls, and lower token costs.

To see how these capabilities come together across end-to-end architectures, read our [Google Cloud Apache Spark in the Agentic Era whitepaper](https://services.google.com/fh/files/misc/google_cloud_apache_spark_whitepaper.pdf). The whitepaper walks through practical design patterns for multimodal data preparation, large-scale feature engineering, and connecting AI agents to governed Apache Iceberg tables through MCP tools and Knowledge Catalog.

Managed Service for Apache Spark 4.0 is available for serverless and managed cluster deployments across all Managed Service for Apache Spark regions. The fastest way to try version 4.0 is the [Data Agent Kit](https://docs.cloud.google.com/data-cloud-extension): install it in VS Code, Claude Code, or Antigravity to build and run Apache Spark workloads on version 4.0 with AI agents right from your IDE. You can also start from the new Overview page in the [Google Cloud console](https://console.cloud.google.com/dataproc), which walks you from code to a running job.

To learn more, read the Managed Service for Apache Spark 4.0 documentation for [serverless deployments and managed clusters](https://docs.cloud.google.com/managed-spark/docs/concepts/versions/spark-runtime-4.0), and try the hands-on [Lightning Engine codelab](https://codelabs.developers.google.com/codelabs/lightning-engine#0).
