# Google’s TPUv8s for Training and Inference at Hot Chips 2026 > Source: > Published: 2026-08-26 00:15:29+00:00 Google’s rate of development for their tensor processor units (TPUs) has been nothing short of remarkable. The company is already preparing their [eighth generation of TPUs](https://www.servethehome.com/google-tpu-8i-for-inference-and-tpu-8t-for-training-announced/), which were announced a bit earlier this year. The TPUv8 family includes both the TPU 8t for training, as well as the TPU 8i for inference, making Google’s hardware offerings unique among the hyperscalers. Whereas the other operators have been focused solely on developing inference hardware while relying on commodity hardware for training, Google has continued to develop their own hardware for both tasks right up to the current day. This experience, Google has argued in the past, is part of what has given them an edge in this market. And the TPUv8 family is meant to extend that. ## Google’s TPUv8s for Training and Inference at Hot Chips 2026 Besides providing their own internal chip design for inference, the TPU 8i is also notable for being a major consumer of Google’s in-house developed CPU, the [Axion](https://www.servethehome.com/google-axion-for-arm-cloud-compute/). TPU 8i chips are paired with Axion CPUs in a 2-to-1 ratio inside Google’s nodes. Previously, the company was using x86 CPUs for this task, so the switch to Axion has both moved Google to their own hardware and has moved them over to the Arm architecture in the process. Briefly, here is a quick look at the history of Google’s TPU development, which is now on their eighth generation. The first TPU was a PCIe card optimized for inference. The next TPU was a design optimized for training with shared memory. The company has (roughly) alternated between inference-optimized chips and training-optimized chips. This has fed into each other, as roughly one-third of the forward pass on training is also a type of inference. Over the years, Google has been able to increase performance by a million-fold. At this point Google is also designing these chips to build for the world. Rather than just building for their internal workloads, there is a bit more consideration for cloud customer workloads (though Google’s internal workloads are still very important to them). What’s new in the eighth-gen TPUs? This year, they are doing two chips in a single year rather than alternating. Due to the wider range of applications, they are seeing a greater need for separate chips. MoE in particular imposes new bottlenecks, as it involves a lot more in the way of communication/network traffic than earlier models. Scaling up FLOPS is not enough. This generation of hardware is also better optimized to handle the long context windows of agentic AI. Now for a brief audience participation pop quiz. Looking at two chips (without their names), how can you tell which is which? The chip on the left is smaller with 6 HBM stacks, the one on the right is larger with 8 HBM stacks. A few people think the one on the left is the inference chip. Most people think the one on the right is the inference chip. The answer is that the left chip is for training (8t) and the one on the right is for inference (8i). (Someone has just made the point that everyone has the slides in advance) But the point is that inference needs more HBM per unit of compute. Inference also needs a higher percentage of SRAM to compute. Looking at the cost of inference, the price goes up quickly the lower the latency is. This is the classic point of inference throughput dropping with higher per-user rates. For their own chip, Google has invested in more SRAM, more memory, and more bandwidth for the 8i. The 8i supports multiple topologies. One particular example here is the BoardFly Topology, with is 8 trays of 4 TPUs, with 36 groups in total. Previously, Google has used a 3D Torus. This is more optimal for training than it is for inference. Fly networks in general have lower latency, which is important for inference. The BoardFly network for 8i has a maximum of 7 hops, versus 16 hops for the torus. The TPU8i design is optimized to perform in-network collectives efficiently. This processing is handled in the ICI I/O die near the edge of the chip next to the networking hardware, rather than processing it in the compute die. This saves the time to transfer to the compute die (including the time to physically move the data) as well as eliminating the HBM accesses. At a high level, the 8i is similar to previous TPU inference designs. Each enclosed box is a compute die, while the purple box is the I/O die. Now shifting gears to the training side of TPUv8 with the TPU 8t chip. No one wants to build an optimized training solution. Training is hard, and Google wants to take it on. Google has been very successful in this regard, with existing products like [Ironwood](https://www.servethehome.com/this-is-the-google-tpu-v7-ironwood-chip/) in use today with customers. Google doesn’t like to make compromises here, hence the separate training and inference chips. Dark silicon eats up precious area and parts of the power budget. Training requires lot of FLOPS, and lots of connectivity. Which is where the focus has been in the design of the chip. Looking at the big picture, the 8t superpod will have 9600 chips, with access to an aggregate 2PB of shared HBM memory. In terms of compute, this is 121 EFLOPS of FP4 compute. In terms of energy efficiency, this is around twice the perf-per-watt as the TPUv7 Ironwood. Not all functions map to FP4, however. So while FP4 is very useful to have, the chip needs to efficiently handled mix precision training recopies, which is how Google has been able to reach that 2x perf-per-watt figure. Here’s a look at a 9600 chip scale-up domain via the use of Google’s optical circuit switch (OCS). It provides both redundancy and reliability. OCS can dynamically configure slices from within the larger pod. Any slice size or shape can be created. OCS also handles bad chips, dropping chips when they go bad, allocating a spare node, restoring from a checkpoint, and then moving on. In previous generation TPUs, Google used their overall data center network. That changes in 8t, where they introduce the dedicated Virgo network architecture. This supports 134K TPUs in a single domain, for 47 Pbits/second of bandwidth. Virgo uses a two layer switching topology. As always, RAS is a huge priority for Google. The working assumption is that the chips will be flaky, so the architecture needs to be able to account for this and work around failed chips. One new RAS feature on this generation is in-field unit testing, which runs during idle cycles to test chips to see if any part of them has failed. Failure rates also increase exponentially with temperatures, which means cooling is critical for Google. The Arrhenius equation says that chemical degradation (and semicon wear) roughly doubles for every 10-15C increase in operating temperature. Here is the block diagram of the 8t, which is very similar to past designs. Only a few training-specific features have been changed. And a look at a complete TPU 8t tray, with 4 TPUs per tray. This is the first time that Google is liquid cooling their optics as well. Previously those were air-cooled. Building up from that, we have the 8t rack. There are 300 of them in Google’s superpod. Google is also using AI to help design their TPUs. These AI models have been able to trim the power usage of TPU 8t, as well as saving area on various bits of both the training and inference chips, allowing Google to pack in more computational cores overall. And briefly on the software front, existing codebases are fully compatible with the TPUv8 generation. Though using Google’s custom kernel language, Pallas, allows writing hardware-aware kernels directly in Python will deliver the best possible performance on both 8t and 8i. Google has been building training and inference chips over the last decade. They are accelerating it this time by doing both together. But their larger design philosophy has not changed, employing a strong co-design link between their software (compiler) and hardware developers in order to avoid doing any more than they have to at runtime.