{"slug": "cloud-pathways-a-distributed-ml-runtime-for-massive-accelerator-fleets-now-ga", "title": "Cloud Pathways, a distributed ML runtime for massive accelerator fleets, now GA", "summary": "Google Cloud announced the general availability of Cloud Pathways, its distributed machine-learning runtime for large accelerator fleets, on Google Kubernetes Engine as a core pillar of the Google Cloud AI Hypercomputer. Google said leading AI companies and global enterprises processed more than 130 million Cloud TPU chip-hours using Pathways on GKE over the last 90 days, and the company also launched a Preview integration with Cluster Director to extend Cloud Pathways to Slurm clusters. The runtime replaces multi-controller JAX management with a single-controller architecture built on the open-source Interim Framework Runtime (IFRT), letting one JAX client process orchestrate multiple TPU slices spanning thousands of chips.", "body_md": "Today, we are thrilled to announce the **general availability (GA)** of [**Cloud Pathways**](https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/pathways-intro) (** Pathways on Cloud**) on Google Kubernetes Engine (GKE) as a core runtime pillar of the Google Cloud **AI Hypercomputer**.\n\nOriginally engineered by Google DeepMind to train and serve Google's frontier models — including Gemini — Pathways is our distributed machine-learning runtime designed for massive, heterogeneous accelerator fleets. In fact, over the last 90 days alone, leading AI companies and global enterprises have processed **more than 130 million Cloud TPU chip-hours** using Pathways on GKE, across a broad range of TPU generations.\n\nWith Cloud Pathways, every Google Cloud customer can use the same single-controller runtime, colocated Python sidecars, and elastic fault tolerance that power Google's largest AI workloads. We are also pleased to announce a [Preview integration with Cluster Director](https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/create-cluster-director-cluster) to extend Cloud Pathways to Slurm clusters, supporting unified multi-slice JAX execution, elastic training, and interactive notebooks.\n\nAs AI workloads scale across thousands of accelerators, spanning multi-slice pre-training, disaggregated inference, and multi-program reinforcement learning (RL) loops, traditional multi-controller setups become harder to program and manage. In default multi-controller JAX, every TPU host runs its own separate Python interpreter with only a localized view of its connected devices and executes the same SPMD program in lockstep, making heterogeneous MPMD workflows and cross-slice resource management complex and brittle.\n\n**Cloud Pathways** replaces multi-controller management with a unified **single-controller architecture** built on the open-source [Interim Framework Runtime (IFRT)](https://github.com/openxla/xla/blob/main/xla/python/ifrt/README.md). By setting `JAX_PLATFORMS=proxy` with [`pathwaysutils`](https://github.com/AI-Hypercomputer/pathways-utils), a single JAX client process views and orchestrates multiple large TPU slices, spanning thousands of TPU chips as if they were all connected locally.\n\n**Learn more about Pathways in our latest deep-dive blog** **here.**\n\nCloud Pathways has the following features:\n\n**Single-controller orchestration:** In Cloud Pathways, a single controller manages graph evaluation, persistent Cloud Storage compilation caching, and asynchronous XLA HLO dispatch across TPU slices.\n\n**Colocated Python sidecars:** Client-side **colocated Python sidecar containers** (with shared-memory support) let you run compute-intensive Python logic such as distributed data loading and checkpointing directly on TPU hosts.\n\n**Resilient and elastic training:** Cloud Pathways decouples job survival from individual slice interruptions with **elastic training** (`pathwaysutils`) using host-memory snapshots to dynamically reshard training down to N-1 slices during hardware failures without crashing the client.\n\n**Interactive mode and Shared Pathways Service:** Run, inspect, and debug JAX workloads interactively from a notebook or terminal using [interactive mode](https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/pathways-interactive-mode), or share a long-running pool of Cloud TPU slices on demand across multiple users and jobs with the [Shared Pathways Service.](https://github.com/AI-Hypercomputer/pathways-utils/tree/main/pathwaysutils/experimental/shared_pathways_service)\n\nFrontier AI labs, quantitative research firms, and global consumer platforms have adopted Cloud Pathways to run mission-critical production workloads on GKE, including:\n\n**Hyperscale multi-host inference:** One of the world's largest consumer technology companies runs production batch and reinforcement learning (RL) inference on Cloud Pathways across tens of thousands of **TPU v5p**, **Trillium (v6e)**, and **Ironwood (TPU7x)** chips, consuming over **100 million chip-hours per quarter,** and co-designed elasticity primitives with Google to expand into pre-training.\n\n**Elastic pre-training and multi-host RL for scientific foundation models:** Google Cloud customer **ZenteiQ** uses Cloud Pathways on GKE with Cluster Toolkit (`gcluster`), colocated Python sidecars, and host snapshotting to scale elastic pre-training (`MaxText`) and single-controller reinforcement learning (` MaxText` + `Tunix`) across multi-slice Trillium (` v6e`) clusters for its scientific foundation models.\n\n\"Running multi-day pre-training and post-training jobs on high-cost accelerators requires reliable continuity through both unexpected hardware faults and planned maintenance windows. By running Cloud TPUs and MaxText with Cloud Pathways on the Google Cloud AI Hypercomputer stack, ZenteiQ relies on a proven controller that monitors our TPU topology and dynamically re-meshes and resizes the workload whenever a chip or slice goes offline. Combining Pathways with host snapshotting in production pushes recovery time down to a minimum when disruptions occur, keeping our available TPU capacity fully utilized with very little stall time. Our ML engineers no longer have to spend their hours watching job health or managing manual restarts, and can stay focused on their research workloads.\" - **Prof. Sashikumaar Ganesan, Founder, ZenteiQ**\n\nCloud Pathways is **generally available today** in all Google Cloud regions supporting **Cloud TPU v5e**, **TPU v5p**, **Trillium (TPU v6e)**, and **Ironwood (TPU 7x)** on GKE, with versioned container images (`us-docker.pkg.dev/cloud-tpu-v2-images/cloud-pathways`) supporting JAX `0.10.1+`. Cloud Pathways is also available in Preview on [Slurm TPU clusters in Cluster Director](https://cloud.google.com/cluster-director/docs/tpu-machines).\n\nTo start building with Cloud Pathways:\n\n**Read the overview:** Explore the [Introduction to Pathways on Cloud](https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/pathways-intro) in the AI Hypercomputer documentation.\n\n**Deploy your cluster:** \n\nGKE: Follow the step-by-step guide to [Create a GKE cluster with Pathways](https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/create-gke-cluster) using Cluster Toolkit (`gcluster`) or the `gcloud` CLI.\n\n**Slurm:** Deploy Pathways via Cluster Director using the [step-by-step guide](https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/create-cluster-director-cluster).\n\n**Run your workloads:** Dive into our production guides for [Batch workloads](https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/batch-workload), [Interactive notebook mode](https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/pathways-interactive-mode), [Resilient & elastic training](https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/resilient-training), [Multihost inference](https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/multihost-inference), [Porting JAX workloads](https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/porting-jax-workloads), and [Troubleshooting](https://docs.cloud.google.com/ai-hypercomputer/docs/workloads/pathways-on-cloud/troubleshooting-pathways).", "url": "https://wpnews.pro/news/cloud-pathways-a-distributed-ml-runtime-for-massive-accelerator-fleets-now-ga", "canonical_source": "https://cloud.google.com/blog/topics/ai-infrastructure/cloud-pathways-a-distributed-ml-runtime-is-ga/", "published_at": "2026-11-09 13:00:00+00:00", "updated_at": "2026-11-09 13:20:26.516859+00:00", "lang": "en", "topics": ["ai-infrastructure", "machine-learning", "ai-chips", "mlops", "developer-tools"], "entities": ["Google Cloud", "Cloud Pathways", "Google Kubernetes Engine", "Google DeepMind", "Gemini", "Interim Framework Runtime", "Cluster Director", "JAX"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/cloud-pathways-a-distributed-ml-runtime-for-massive-accelerator-fleets-now-ga", "markdown": "https://wpnews.pro/news/cloud-pathways-a-distributed-ml-runtime-for-massive-accelerator-fleets-now-ga.md", "text": "https://wpnews.pro/news/cloud-pathways-a-distributed-ml-runtime-for-massive-accelerator-fleets-now-ga.txt", "jsonld": "https://wpnews.pro/news/cloud-pathways-a-distributed-ml-runtime-for-massive-accelerator-fleets-now-ga.jsonld"}}