# Google Split Its TPU in Two Because Agents Broke Inference > Source: > Published: 2026-07-25 13:09:15+00:00 [Cloud & Infra](https://sourcefeed.dev/c/cloud)Article # Google Split Its TPU in Two Because Agents Broke Inference Cloud Next 2026's bifurcated silicon, flattened networks, and managed sandboxes reveal what agentic workloads actually demand. [Ji-ho Choi](https://sourcefeed.dev/u/jiho_choi) For seven generations, Google built one TPU and asked it to do everything. At Cloud Next 2026 it stopped: the eighth generation splits into [TPU 8t](https://cloud.google.com/blog/products/compute/tpu-8t-and-tpu-8i-technical-deep-dive), a training chip, and TPU 8i, an inference chip — different dies, different memory configurations, even different network topologies. That bifurcation, more than any single spec, tells you what Google thinks is about to happen to its data centers. And the culprit it keeps naming is the AI agent. The demand math is stark. Sundar Pichai said at I/O in May that Google now processes 3.2 quadrillion tokens a month, a sevenfold jump in a year. Amin Vahdat, who runs AI infrastructure at Google, has put a multiplier on the next phase: agentic workloads generate 50 to 100 times more inference transactions than a single chat turn, because one user request fans out into planning steps, tool calls, retries, and re-reads of an ever-growing context. Whether or not you believe agents will run the economy, Google is spending like it does — capex guidance for 2026 sits at $175–185 billion, roughly double last year. ## Agents break the assumptions inference hardware was built on Classic LLM serving is throughput-friendly: batch a thousand independent prompts, keep the matrix units fed, amortize everything. Agents wreck that in three specific ways. First, they're memory-bound, not compute-bound. An agent carrying a long context through a multi-step task drags a huge KV cache with it, and decode speed becomes a function of how fast you can stream that cache, not how many FLOPS you have. TPU 8i's spec sheet reads like an admission of this: 288 GB of HBM per chip (Ironwood had 192), higher memory bandwidth than its own training sibling, and 384 MB of on-chip SRAM — triple the previous generation — explicitly to keep more KV cache on-die instead of round-tripping to HBM. Second, agents are latency-serial. A chain-of-thought loop that calls a tool, reads the result, and reasons again can't be batched away; every network hop in the pod is on the critical path. So 8i abandons the 3D torus that TPUs have used for years in favor of a high-radix topology Google calls Boardfly, cutting worst-case hops from 16 to 7. The training chip keeps the torus, because training traffic is bulk-synchronous and doesn't care. Two chips, two networks, one workload split. Third, a surprising share of agent work isn't matrix math at all. Orchestration, API calls, sandboxed code execution, memory management — that's CPU territory. Google's answer is its Arm-based Axion line and, on top of it, [Agent Sandbox on GKE](https://cloud.google.com/blog/products/containers-kubernetes/bringing-you-agent-sandbox-on-gke-and-agent-substrate): [gVisor](https://gvisor.dev/)-isolated execution environments for untrusted agent code, spinning up at up to 300 sandboxes a second per cluster. That's the layer teams currently hand-roll with Firecracker microVMs or container hacks, offered as a managed primitive. Above the pod, a new scale-out fabric called Virgo flattens the traditional three-layer data center network into two, stitching 134,000 TPUs into one fabric and — via the Pathways runtime, now available to cloud customers for the first time — over a million chips across sites into a single training job. Virgo isn't TPU-only: it also fronts NVIDIA-based clusters up to 960,000 GPUs across sites, which says something about how seriously Google takes hedging. ## This isn't a Google quirk — it's the whole industry's turn The strongest evidence that specialized inference silicon is a real shift rather than Cloud Next stagecraft is that NVIDIA got there first, from the opposite direction. Its [Rubin CPX](https://nvidianews.nvidia.com/news/nvidia-unveils-rubin-cpx-a-new-class-of-gpu-designed-for-massive-context-inference), announced back in September 2025, is a GPU built specifically for the compute-heavy prefill phase of massive-context inference, disaggregated from the memory-heavy decode phase. Google is splitting training from inference; NVIDIA is splitting inference itself in half. Different cuts, same diagnosis: one general-purpose accelerator per workload was a compromise, and at $100-billion capex scale, compromises are the most expensive line item there is. There's precedent for how this ends. GPUs displaced CPUs for neural nets because generality lost to specialization once the workload stabilized. Agentic inference — long contexts, sparse MoE models, serial reasoning — has now stabilized enough to justify its own silicon. Expect AWS and Microsoft to follow with inference-skewed variants of Trainium and Maia; the era of the single do-everything AI chip is closing. ## What you can actually use, and when Temper the excitement with the calendar. The 8th-gen TPUs are announced, not rentable: Google has published specs and an interest form but no GA date, and supply-chain reporting points to volume in 2027. What you can get today is [Ironwood](https://cloud.google.com/tpu), which went GA at Cloud Next. The forward-looking claims — up to 2.7× better training price-performance for 8t, up to 80% better inference price-performance for 8i on low-latency MoE serving — are Google's own projections against Ironwood, so treat them as directional until third-party benchmarks exist. The nearer-term, more practical shifts for anyone running agents: **The TPU software wall is lower than it was.** Native PyTorch support is in preview, vLLM runs on TPU, and Pathways handles multi-host inference with dynamic scaling. If your serving stack is vLLM-on-CUDA, a TPU price-performance experiment is now a realistic afternoon project rather than a JAX rewrite.**Stop putting orchestration on GPUs.** The scaffolding around an agent — tool routing, retries, state — runs fine on Axion N4A instances, which Google claims roughly 2× price-performance over comparable x86 VMs. Splitting the agent loop (CPU) from the model call (accelerator) is the cost optimization most teams haven't done yet.**Managed sandboxing changes the build-vs-buy math.** If you're maintaining your own microVM pool for agent code execution, GKE's Agent Sandbox is the first hyperscaler-native alternative worth evaluating — with the usual caveat that it deepens your Google Cloud commitment. The honest caveat on the whole edifice: the 50–100× inference multiplier is Google's demand forecast, and it's also the justification for Google's spending. Those numbers come from the same building. But the engineering constraints agents impose — memory-bound decode, serial latency, CPU-heavy orchestration — are real today at any scale, and the fixes Google shipped for itself are the same ones you'll be making in your own stack, just with smaller numbers. The hyperscalers have decided agents are a different workload. On the evidence of the silicon, they're right. ## Sources & further reading - [Google reengineers data center hardware for AI agents](https://dev.to/vpodk/google-reengineers-data-center-hardware-for-ai-agents-ba8)— dev.to - [TPU 8t and TPU 8i technical deep dive](https://cloud.google.com/blog/products/compute/tpu-8t-and-tpu-8i-technical-deep-dive)— cloud.google.com - [Google launches Ironwood TPU and previews eighth-gen split into training and inference chips](https://thenextweb.com/news/google-ironwood-tpu-inference-cloud-next)— thenextweb.com - [Google Bolsters AI Hypercomputer with New TPU Chips, Virgo Interconnect, Speedier Lustre](https://www.hpcwire.com/2026/04/22/google-bolsters-ai-hypercomputer-with-new-tpu-chips-virgo-interconnect-speedier-lustre/)— hpcwire.com - [Q&A: Google's AI and computing chief talks about its shapeshifting data centers](https://www.networkworld.com/article/4200625/qa-googles-ai-and-computing-chief-talks-about-its-shapeshifting-data-centers.html)— networkworld.com - [Google touts its tokenmaxxing and capex spending amid AI orgy](https://www.theregister.com/ai-ml/2026/05/19/google-touts-tokenmaxxing-huge-capex-and-ai-agents-at-i/o/5242983)— theregister.com - [Bringing you Agent Sandbox on GKE and Agent Substrate](https://cloud.google.com/blog/products/containers-kubernetes/bringing-you-agent-sandbox-on-gke-and-agent-substrate)— cloud.google.com [Ji-ho Choi](https://sourcefeed.dev/u/jiho_choi)· Security & Cloud Editor Ji-ho covers the increasingly tangled overlap between cloud architecture and security, drawing on a background as a penetration tester to keep his reporting grounded in real-world attack paths. He never lets a vendor claim go unquestioned and insists that every buzzword come with a proof of concept. ## Discussion 0 No comments yet Be the first to weigh in.