Google designed TPUs from the ground up to be exceptionally efficient for machine learning workloads, delivering top-tier performance per total cost of ownership (TCO). Because TPUs trade general-purpose hardware scheduling for wide systolic arrays, dedicated SparseCore engines, and custom interconnect fabrics, adhering to specific architectural best practices is essential to achieve maximum performance. To help developers get the most out of TPUs, we are launching the open-source AI-Hypercomputer/tpu-codesign guide. This post highlights its key principles—spanning shape alignment, architectural choices like LatentMoE, and parallelism strategies—to accelerate LLMs, Mixture-of-Experts (MoE) models, and other hardware-intensive workloads on TPUs.
Performance tuning starts with model definition. Architectural choices made before the first training step set permanent ceilings on MFU (Model FLOP Utilization), cost, and serving latency. This guide gives concrete, hardware-grounded design rules for teams building or adapting models for TPUs, organized into three groups:
How to use this guide. The shape rules apply to anyone designing a new model. If you are adapting an existing model rather than designing one, the Expressing Parallelism section applies without touching the checkpoint. The general rules hold for any large-matmul architecture (dense LLMs, multimodal, and world models), and the companion AI-Hypercomputer/tpu-codesign repository grows model- and workload-specific sections over time.
Eighth-generation TPUs. These principles carry forward to TPU 8t and TPU 8i and to the newly announced TPU 8tx and TPU 8i Max. The reasoning stays the same and the thresholds move with the hardware. TPU 8t and TPU 8i both add native FP4. For training, TPU 8t also brings more balanced Vector Processing Unit (VPU) scaling. For serving, TPU 8i adds the Boardfly topology, a Collectives Acceleration Engine (CAE), and 3x more on-chip SRAM. We note the effects inline in Principles 5–8, and follow-up posts will cover eighth-generation designs in depth.
Measured evidence. Alongside the published sources, notes marked Measured carry results from our own Ironwood profiling program: open 20–30B MoEs (a Qwen3-30B-derived LatentMoE variant; GPT-OSS-20B) plus dense controls, single slice (except for the 512-chip TPU v7x and 1,024-chip TPU v4 ablations in Principle 4), BF16, training shapes up to 32K sequence, all against tuned-systems baselines. Treat the trends and rankings as transferable, the exact percentages as config-specific.
TPUs are purpose-built custom ML ASICs designed from the ground up to deliver state-of-the-art performance per total cost of ownership (TCO). While their architectural choices—such as wide 256×256 systolic arrays, software-managed VMEM, dedicated SparseCore engines, and direct multi-dimensional ICI fabrics—enable exceptional efficiency, they inherently create specific structural constraints and programming patterns that must be carefully followed to unlock maximal performance. Reaching peak efficiency requires aligning model shapes, routing, and parallelism directly with these underlying hardware characteristics:
Make d_model, FFN and expert intermediate widths ( d_ff), latent widths, attention projections, and padded vocabulary sizes (used as contracting or output dimensions) multiples of 256 to match the 256×256 MXU systolic array. Token/batch axes and counts such as heads, layers, and experts are exempt. The most frequently missed dimension is head_dim: at 64 or 128, flash-attention QK products leave the MXU ≥ 50% idle. Reshape to head_dim = 256 with proportionally fewer Q and KV heads, preserving total projection widths and logical KV cache bytes (iso-quality in our ablations).
Measured: Reshaping Qwen3-30B MoE attention from 32 × 128 to 16 × 256 (identical parameters and FLOPs) sped up end-to-end training, with gains widening at longer context:
These rules apply for non-quadratic attention as well, notably GDN ablations from Yang et al. 2025 also noted at par or better quality for head dimension 256 for their Gated DeltaNet Attention.
XLA compiles each program for exact tensor shapes, so every buffer has a fixed address and every loop a known trip count, and a new shape means a recompilation and a host stall. What about dynamic shapes? Dynamic workloads fit this model: the trick is to separate the shape of a buffer from the extent of the data inside it.
Instead of padding to the largest group or the longest sequence, we keep the buffers at fixed shapes (tokens sorted by expert and padded only at the tail, the KV cache as a fixed pool of pages) and describe the ragged structure with a few small integer arrays (group sizes and offsets; KV lengths, the page table, cumulative query lengths). Pallas copies these into the TensorCore's scalar memory before the grid starts (PrefetchScalarGridSpec), and the kernel reads them to choose the next tile and bound its loops, with a mask covering a tile that straddles a group boundary or a partly filled last page. The data dependence becomes an address and a trip count instead of a shape, so the compiled program is the same for every routing and every batch composition and the kernel runs at the unpadded FLOP count.
Standard EP dispatches every routed token at full d_model width, making EP + FSDP slower than plain FSDP in our Ironwood benchmarks. LatentMoE projects routed tokens from d_model down to latent width ℓ (d/ℓ ≈ 2–4×) before dispatch, executes experts in latent space, and projects back up after combine (introduced by Nemotron 3 and adopted by 962B MAI-Base-1 and 2.8T Kimi K3):
m = d_ff, latent compression cuts small-batch expert weight reads (∝ d × m) and throughput-bound all-to-all dispatch (∝ K × d) by d/ℓ. Reinvest the saved bandwidth into more active experts while keeping K_new × ℓ ≤ K_old × d_model, raising the nonlinearity budget K × m at constant traffic (for example, Nemotron 3's ℓ = 1024, N = 512, top-22 beat d = 4096, N = 128, top-6 at matched active and total parameters).d_model before compressing; keep ℓ a multiple of 256; dispatch in FP8 while keeping lightweight up/down projections in BF16; and place EP on one short 3D torus axis.
Measured: On a 30B / 128-expert MoE, an ℓ = 768 (2.67×) LatentMoE cut on-device permute time 2.8×, lifted end-to-end training throughput +14% tokens/s (322 vs 270 TFLOP/s/chip, or +19%, since the latent projections add ~5% FLOPs per token; tokens stay ℓ-wide past the down-projection, so on-device permute and combine shrink alongside EP all-to-all), and freed ~25 GB/device of activation HBM when remat checkpoints were anchored on the compressed latent. While current MoE recipes in tpu-recipes run FSDP (+ TP), LatentMoE + FP8 dispatch makes EP roofline-viable when per-device batch is too small to hide FSDP weight all-gathers. Under both FSDP and EP, avoid over-fragmenting experts: small tokens × K / N group sizes starve MXU tiles.
When candidate MoE layouts match on quality, pick the one with higher compute intensity at a similar active-parameter budget. A larger shared expert (wider d_shared) shifts work into dense, communication-free GEMMs that overlap EP dispatch/combine in training and amortize weight reads across the decode batch. Fewer, wider routed experts (halving N and K while doubling d_ff at fixed sparsity K/N) cut dispatch traffic in half and double per-expert GEMM fill. Optimal (N, K) grows with model size, and over-coarsening hurts routing diversity. Ablate expert granularity and the shared-to-routed split at your target scale rather than copying GPU co-designed models, since the TPU optimum is more shared-heavy.
Measured: On a 63B LatentMoE (ℓ = 768, K/N = 1/32, 8K context, 512 TPU v7x chips at EP = 128), halving ( N, K) from (512, 16) to (256, 8) while doubling d_ff and the shared expert lifted training throughput +57% tok/s and held lower loss across 120.8B tokens:
Low-concurrency decode streams model weights every token (~300 concurrent tokens per weight shard are needed in dense FP8 to reach compute roofline). Combine two architectural levers with explicit concurrency sizing:
per-chip weight bytes ÷ HBM bandwidth (for example, when the batch touches all experts, Qwen3.5-397B in FP8 across 4 chips streams ~100 GB/chip, setting a ~13.5 ms weight-read floor before KV), while max batch is capped by (remaining HBM) ÷ (allocated KV/state per sequence). For MoEs, weight traffic scales with the E × [1 - (1 - K/E)^B] under uniform routing); profile busiest-rank skew and replicate hot experts (
Norms, activations, masking, routing math and quantization scale applies run on the Vector Processing Unit (VPU), which is roughly two orders of magnitude below the MXU in throughput and executes in sequence with the matmuls unless XLA fuses it into an adjacent GEMM or we move it off the TensorCore. Therefore we drop reshapes and transposes that carry no quality, standardize on fusable RMSNorm and SwiGLU so the elementwise work folds into the neighbouring GEMM, and offload the MoE sort, gather and scatter to the SparseCore.
Note that fused VPU work is free only while its cycle count fits inside the MXU window it is fused into. A quick illustration of this concept is block-scaled fp8 GEMM.
With a scale per block of B elements along the contraction axis, the MXU emits one partial sum per output element per block and the VPU rescales each one (about three vector ops), so per output element the MXU does 2K flops and the VPU does 3·ceil(K/B) ops. The rescale hides when B ≥ 1.5 × r_mxu / r_vpu, which on v7x (fp8 peak 4,614 TFLOPS against a 19.2 TFLOPS vector peak, a ratio near 240) puts the threshold near 340. In simulation on a DeepSeek-V3 expert shape, block 512 runs at the per-tensor time, block 256 is VPU-bound at 1.2x and block 128 at 2.4x. Achieved MXU efficiency is lower than peak, which lowers the effective ratio and can make the smaller blocks workable; the way to decide is to measure the GEMM at each block instead of using peak ratio. This threshold is specific to Ironwood. TPU 8t is designed with specialized support for scaling so quantization for MXFP4 and MXFP8 formats has no overhead. Re-measure the block-size threshold there instead of reusing 340. The rule itself carries forward: keep vector work hidden inside a matmul window.
For workloads dominated by BF16-sensitive matmuls (most notably long-context quadratic attention where direct FP8 degrades accuracy), native single-pass BF16 leaves MXU capacity unused whenever hardware low-precision MXU throughput exceeds native BF16 throughput by more than the pass count. Use the multi-pass BF16 algorithm to execute BF16-accurate matmuls via multiple passes over the higher-throughput lower-precision MXU pipeline without losing numerical fidelity. This matters more as low-precision throughput pulls ahead of BF16. TPU 8t adds native FP4, which doubles MXU throughput. On Ironwood the FP8 to BF16 ratio is exactly 2:1, so two or more passes cannot beat native BF16. Assign communication-heavy parallel axes to the fastest physical links and keep logical mesh factors aligned with torus dimensions:
Minimum-degree CP matched to kernel trade-offs: All-gather CP is simplest (though its all-gathers can sit exposed) when paired with zigzag load balancing (context_parallel_load_balance in MaxText) so causal masking does not idle tail ranks; Ring CP overlaps collectives but trades off SplashAttention chunk efficiency; Ulysses A2A CP avoids causal skew but exposes all-to-alls and caps at head count.
Start every project with the quality-safe foundation: 256-aligned dimensions (1), static shapes with dropless routing (2), one parallelism plan mapped to physical links (8), wall-clock ablations on a tuned stack, and published reference targets (both in Measuring what matters below). For MoEs, add LatentMoE (3) after an ablation to cut routing/dispatch traffic and decode weight reads. Then prioritize by bottleneck regime:
d_ff, SparseCore offload, 1D torus EP).
Google engineered TPUs from the ground up as custom ML ASICs to deliver top-tier performance per total cost of ownership (TCO). Unlocking that potential requires bridging the gap between hardware architecture and model design. By standardizing on 256-aligned shapes, leveraging LatentMoE, and mapping parallel axes directly to physical links, engineering teams can maximize TPU efficiency across both training and serving workloads.
A reference blueprint is only as good as the loop used to validate it. Always evaluate architectural changes in wall-clock time on a tuned software stack rather than paper FLOPs: in our Ironwood benchmarks, pure systems tuning (SparseCore offload, MXU-aligned tiles, and remat tuning) lifted an open 30B MoE from 12% to 23% MFU before any model change, and published Ironwood baselines show 256-aligned MoEs (DeepSeek-V3, Qwen3-235B at 608–630 BF16 TFLOP/s/chip) running at roughly 2× the throughput of unaligned GPT-OSS-120B (330 TFLOP/s/chip). For the 6-signal XProf (profiler=xplane) triage checklist, roofline and HBM budgeting rules, wall-clock ablation methodology, published Ironwood reference targets, and ongoing updates as new TPU generations and kernels land, explore the living diagnostics guide and companion patterns in the open-source AI-Hypercomputer/tpu-codesign repository.