Cloud Pathways, a distributed ML runtime for massive accelerator fleets, now GA

Google Cloud announced the general availability of Cloud Pathways, its distributed machine-learning runtime for large accelerator fleets, on Google Kubernetes Engine as a core pillar of the Google Cloud AI Hypercomputer. Google said leading AI companies and global enterprises processed more than 130 million Cloud TPU chip-hours using Pathways on GKE over the last 90 days, and the company also launched a Preview integration with Cluster Director to extend Cloud Pathways to Slurm clusters. The runtime replaces multi-controller JAX management with a single-controller architecture built on the open-source Interim Framework Runtime (IFRT), letting one JAX client process orchestrate multiple TPU slices spanning thousands of chips.

Today, we are thrilled to announce the general availability GA of Cloud Pathways https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/pathways-intro Pathways on Cloud on Google Kubernetes Engine GKE as a core runtime pillar of the Google Cloud AI Hypercomputer . Originally engineered by Google DeepMind to train and serve Google's frontier models — including Gemini — Pathways is our distributed machine-learning runtime designed for massive, heterogeneous accelerator fleets. In fact, over the last 90 days alone, leading AI companies and global enterprises have processed more than 130 million Cloud TPU chip-hours using Pathways on GKE, across a broad range of TPU generations. With Cloud Pathways, every Google Cloud customer can use the same single-controller runtime, colocated Python sidecars, and elastic fault tolerance that power Google's largest AI workloads. We are also pleased to announce a Preview integration with Cluster Director https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/create-cluster-director-cluster to extend Cloud Pathways to Slurm clusters, supporting unified multi-slice JAX execution, elastic training, and interactive notebooks. As AI workloads scale across thousands of accelerators, spanning multi-slice pre-training, disaggregated inference, and multi-program reinforcement learning RL loops, traditional multi-controller setups become harder to program and manage. In default multi-controller JAX, every TPU host runs its own separate Python interpreter with only a localized view of its connected devices and executes the same SPMD program in lockstep, making heterogeneous MPMD workflows and cross-slice resource management complex and brittle. Cloud Pathways replaces multi-controller management with a unified single-controller architecture built on the open-source Interim Framework Runtime IFRT https://github.com/openxla/xla/blob/main/xla/python/ifrt/README.md . By setting JAX PLATFORMS=proxy with pathwaysutils https://github.com/AI-Hypercomputer/pathways-utils , a single JAX client process views and orchestrates multiple large TPU slices, spanning thousands of TPU chips as if they were all connected locally. Learn more about Pathways in our latest deep-dive blog here. Cloud Pathways has the following features: Single-controller orchestration: In Cloud Pathways, a single controller manages graph evaluation, persistent Cloud Storage compilation caching, and asynchronous XLA HLO dispatch across TPU slices. Colocated Python sidecars: Client-side colocated Python sidecar containers with shared-memory support let you run compute-intensive Python logic such as distributed data loading and checkpointing directly on TPU hosts. Resilient and elastic training: Cloud Pathways decouples job survival from individual slice interruptions with elastic training pathwaysutils using host-memory snapshots to dynamically reshard training down to N-1 slices during hardware failures without crashing the client. Interactive mode and Shared Pathways Service: Run, inspect, and debug JAX workloads interactively from a notebook or terminal using interactive mode https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/pathways-interactive-mode , or share a long-running pool of Cloud TPU slices on demand across multiple users and jobs with the Shared Pathways Service. https://github.com/AI-Hypercomputer/pathways-utils/tree/main/pathwaysutils/experimental/shared pathways service Frontier AI labs, quantitative research firms, and global consumer platforms have adopted Cloud Pathways to run mission-critical production workloads on GKE, including: Hyperscale multi-host inference: One of the world's largest consumer technology companies runs production batch and reinforcement learning RL inference on Cloud Pathways across tens of thousands of TPU v5p , Trillium v6e , and Ironwood TPU7x chips, consuming over 100 million chip-hours per quarter, and co-designed elasticity primitives with Google to expand into pre-training. Elastic pre-training and multi-host RL for scientific foundation models: Google Cloud customer ZenteiQ uses Cloud Pathways on GKE with Cluster Toolkit gcluster , colocated Python sidecars, and host snapshotting to scale elastic pre-training MaxText and single-controller reinforcement learning MaxText + Tunix across multi-slice Trillium v6e clusters for its scientific foundation models. "Running multi-day pre-training and post-training jobs on high-cost accelerators requires reliable continuity through both unexpected hardware faults and planned maintenance windows. By running Cloud TPUs and MaxText with Cloud Pathways on the Google Cloud AI Hypercomputer stack, ZenteiQ relies on a proven controller that monitors our TPU topology and dynamically re-meshes and resizes the workload whenever a chip or slice goes offline. Combining Pathways with host snapshotting in production pushes recovery time down to a minimum when disruptions occur, keeping our available TPU capacity fully utilized with very little stall time. Our ML engineers no longer have to spend their hours watching job health or managing manual restarts, and can stay focused on their research workloads." - Prof. Sashikumaar Ganesan, Founder, ZenteiQ Cloud Pathways is generally available today in all Google Cloud regions supporting Cloud TPU v5e , TPU v5p , Trillium TPU v6e , and Ironwood TPU 7x on GKE, with versioned container images us-docker.pkg.dev/cloud-tpu-v2-images/cloud-pathways supporting JAX 0.10.1+ . Cloud Pathways is also available in Preview on Slurm TPU clusters in Cluster Director https://cloud.google.com/cluster-director/docs/tpu-machines . To start building with Cloud Pathways: Read the overview: Explore the Introduction to Pathways on Cloud https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/pathways-intro in the AI Hypercomputer documentation. Deploy your cluster: GKE: Follow the step-by-step guide to Create a GKE cluster with Pathways https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/create-gke-cluster using Cluster Toolkit gcluster or the gcloud CLI. Slurm: Deploy Pathways via Cluster Director using the step-by-step guide https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/create-cluster-director-cluster . Run your workloads: Dive into our production guides for Batch workloads https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/batch-workload , Interactive notebook mode https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/pathways-interactive-mode , Resilient & elastic training https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/resilient-training , Multihost inference https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/multihost-inference , Porting JAX workloads https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/porting-jax-workloads , and Troubleshooting https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/troubleshooting-pathways .