Distributed LLM Training Across Metros: Streaming DiLoCo in MaxText Google DeepMind and Google Research detailed Streaming DiLoCo, a distributed optimization algorithm implemented in MaxText, Google's open-source JAX-based LLM training framework, that trains models across geographically distributed datacenters over wide-area networks. Streaming DiLoCo partitions model parameters into P disjoint fragments so every fragment synchronizes exactly once over H steps, eliminating the periodic full-model transfer stall of vanilla DiLoCo while cutting communication frequency by a factor of H — 1 to 2 orders of magnitude less than standard synchronous training — with comparable convergence and model quality. The approach targets the collapse in Model FLOPS Utilization that standard data parallelism suffers when WAN bandwidth drops and latency rises. Traditionally, training state-of-the-art Large Language Models LLMs relies on a monolithic cluster of accelerators, where all devices are collocated and tightly coupled via high-bandwidth to constantly synchronize gradients. However, physical and cost constraints make scaling a single slice increasingly difficult. To scale further, we can connect multiple compute slices across geographically distributed datacenters. However, when training across multiple datacenters over wide-area networks WAN , standard data-parallel training suffers from severe network stalls because slow WAN transfers cannot be fully overlapped with computation. To address this, the Google DeepMind and Google Research teams developed DiLoCo Distributed Low-Communication , a distributed optimization algorithm designed to make training resilient to these low-bandwidth, high-latency connections. In this blog post, we discuss the core concepts of DiLoCo, how it is implemented in MaxText Google's open-source JAX-based LLM training framework , and the performance benefits it brings to large-scale LLM training on Google Cloud. DiLoCo was introduced by Google DeepMind in the paper DiLoCo: Distributed Low-Communication Training of Language Models https://arxiv.org/abs/2311.08105 . Instead of synchronizing gradients across all devices at every single training step as in standard Data Parallelism , DiLoCo partitions training across M independent compute groups, referred to as learners, and allows each learner to perform many local optimization steps independently before communicating across groups. DiLoCo operates on two levels: H steps, the M learners compute the difference delta between their current weights and the global weights at the start of the interval. The learners average these deltas across the network via a distributed all-reduce and apply an outer optimizer typically SGD with Nesterov momentum to update the global weights, which then serve as the starting point for the next inner-loop interval. Despite communicating significantly less frequently by a factor of H, e.g., 1-2 orders of magnitude less than standard synchronous training, DiLoCo has been shown to achieve comparable convergence and model quality https://arxiv.org/abs/2503.09799v1 . The Vanilla DiLoCo described above reduces total communication volume by a factor of H, however, all learners still transmit their weight deltas for the entire model all at once at every H-th step. On a thin network pipe, that full-model transfer creates a large periodic stall every H steps. This issue is alleviated by Streaming DiLoCo, which is described in the next section. In multi-slice training, communication within a single slice is handled by high-speed Inter-Chip Interconnects ICI . When scaling to multiple slices, synchronization across slices must traverse the Data Center Network DCN or, if the slices are geographically distributed, wide-area networks WAN . Within a single datacenter, DCN bandwidth is high enough that JAX and XLA can often hide standard Data Parallel all-reduce collectives by overlapping gradient communication with backward-pass computation. Across datacenters or congested links, however, bandwidth drops and latency rises sharply. Standard DP stalls at every step waiting for global gradient synchronization, collapsing Model FLOPS Utilization MFU . Streaming DiLoCo https://arxiv.org/abs/2501.18512 eliminates this synchronization stall by partitioning the model's parameters into P disjoint fragments . Over the course of H steps, every single fragment in the model is synchronized exactly once, achieving the exact same outer-loop cadence as vanilla DiLoCo without spiky bandwidth utilization. Using a roofline model similar to the JAX Scaling Book https://jax-ml.github.io/scaling-book/training/ , we can calculate the network bandwidth required to prevent communication stalls for a model with ∣Θ∣ parameters at b bytes per parameter, e.g., b=2 for bfloat16 and per-step compute time tstep=tfwd+tbwd. For example, the standard Data Parallel algorithm transfers 2∣Θ∣b bytes per sync via an all-reduce collective, which is a distributed reduce-scatter where each node reduces a different part, followed by an all-gather so all participants receive all parts. To achieve zero communication stalls, we overlap the reduce-scatter of the gradients with the backward pass, which produces them and takes tbwd time. Therefore, the peak required bandwidth is Bdp ≈ 2∣Θ∣b / tbwd. Worked Example Qwen3-32B in bfloat16 , tstep≈6.2s, P=H=65 : To evaluate distributed efficiency under realistic network bottlenecks, we benchmarked Qwen3-32B 64 decoder layers, partitioned into P = 65 fragments with H = 65 across 2× TPU v5p-32 slices dcn diloco parallelism=2 using built-in Linux traffic-control tc bandwidth shaping to limit inter-slice bandwidth across three network regimes: Unthrottled Line Rate , 30 Gbit/s aggregate Moderate Congestion , and 10 Gbit/s aggregate Severe Congestion . Key Findings: P = 65, H = 65 maintains a flat 5.76 s/step—delivering a 9.57× speedup over DP and eliminating the step- A common question with low-communication training is whether allowing learners to train independently between synchronizations degrades model quality—especially on Mixture-of-Experts MoE architectures, where independent updates could cause expert routers to drift apart. We evaluated Streaming DiLoCo against synchronous Data Parallel pre-training across two MoE architectures in MaxText Qwen3-30B-A3B and Gemma4-26B-A4B over 4,000 training steps: Empirical results confirm that Streaming DiLoCo preserves—and even slightly improves—validation loss: β = 0.9 , outer learning rate η outer between 0.3 and 0.7 across fragment pseudo-gradients stabilizes the global trajectory. MaxText provides a native JAX and Flax NNX implementation of both Vanilla and Streaming DiLoCo—requiring no external central parameter servers. You can enable Streaming DiLoCo in your MaxText YAML configuration or CLI overrides by setting enable diloco: true and enable streaming diloco: true . As a best practice, set both num diloco fragments P and diloco sync period H to N layers + 1 so that exactly one fragment is synchronized per training step Δh = 1 : Enable DiLoCo and Streaming DiLoCo enable diloco: true enable streaming diloco: true Partition into N layers + 1 fragments e.g., 36 layers + 1 embedding/head = 37 for Qwen3-8B, 48 + 1 = 49 for Qwen3-30B-A3B, or 64 + 1 = 65 for Qwen3-32B num diloco fragments: 37 diloco sync period: 37 use sequential layers: false false = interleaved layers across fragments recommended Outer Nesterov SGD optimizer hyperparameters diloco outer lr: 0.7 diloco outer momentum: 0.9 Multi-slice topology: number of DiLoCo learner islands across DCN/WAN dcn diloco parallelism: 2 e.g., 2 TPU slices or set to -1 to infer from num slices Optional: emulate constrained WAN bandwidth on fast clusters via Linux tc dcn bandwidth limit: "10gbit" In MaxText, diloco.py https://github.com/AI-Hypercomputer/maxtext/blob/main/src/maxtext/trainers/diloco/diloco.py splits each global batch across the M learners so each learner processes GBS / M tokens per step. To overlap cross-slice communication with TPU computation in Streaming DiLoCo, weight deltas for the scheduled fragment are offloaded to M -way sharded syncer workers running on the learner host CPUs. Rather than routing through a single central coordinator, the host-CPU syncer shards execute a distributed all-reduce across the network in the background to average the fragment deltas while the TPUs continue running inner-loop training steps. Each syncer shard then applies the outer Nesterov momentum update locally and writes the updated fragment weights back to its colocated TPU learner. To dive deeper or launch your first run: Streaming DiLoCo in MaxText decouples large-scale LLM pre-training from the requirement of a single, monolithic, high-bandwidth fabric. By combining use of an inner and outer optimizer with fine-grained parameter streaming, engineering teams can pool TPU slices across congested datacenter networks or geographically distributed metros—sustaining high hardware utilization without compromising model convergence. Streaming DiLoCo https://maxtext.readthedocs.io/en/maxtext-v0.2.5/tutorials/diloco pretraining.html is available today in MaxText v0.2.5 https://pypi.org/project/maxtext/0.2.5/ on PyPI, the corresponding Docker images https://maxtext.readthedocs.io/en/latest/tutorials/build maxtext.html pre-built-maxtext-docker-images and on GitHub main. Try it today for experimenting with distributed training workloads.