The Current State of AI Chips

NVIDIA and Google are racing to dominate the AI chip market, with NVIDIA's Blackwell and Rubin architectures and Google's TPU v8 and Ironwood v7 leading the shift toward energy-efficient 'tokens per watt' performance, as datacenter power limits become the primary constraint. The article details how these chips balance compute-bound prefill and memory-bound decode phases, with NVIDIA's GB200 NVL72 rack saving ~20 kW per rack via copper cabling and Google's TPUs using weight-stationary systolic arrays.

Hardware Architecture Deep Dive12 min read·August 2026 The Current State of AI Chips: Architectures, Silicon Wars, and the Shift to Watts-Per-Token In 2018, Turing laureates John Hennessy and David Patterson predicted that the death of Dennard scaling would spark a “Cambrian explosion” in domain-specific computer architecture. Eight years later, that explosion has arrived—and the battleground is no longer just theoretical FLOPs, but datacenter thermal ceilings and Tokens per Joule. Today, dozens of distinct silicon architectures compete for AI workloads: programmable SIMT GPUs, compile-time systolic ASICs, wafer-scale SRAM engines, deterministic spatial LPUs, and clean-sheet custom silicon taped out directly by frontier AI labs. Meanwhile, the economic bottleneck has transformed. When a datacenter is capped at a strict 50-megawatt grid connection, raw chip FLOPs don't generate revenue—sustained Tokens Per Megawatt TPS/MW does. Here is the full breakdown of how modern AI silicon actually works, where the tradeoffs lie, and how the hardware landscape is stratifying. 1. The Physics of the Workload: Two Conflicting Regimes Every AI accelerator is an answer to a mathematical dilemma: modern generative models transformers, mixture-of-experts, diffusion models are sequences of massive matrix multiplications GEMMs interleaved with memory-bound vector operations and autoregressive generation. Phase 1: Prefill / TrainingCompute-Bound Large GEMM Computations Stacks entire batches of prompt tokens against weight matrices in parallel. Arithmetic intensity is high hundreds of FLOPs per byte loaded . The hardware is limited purely by raw tensor core density. Phase 2: Autoregressive DecodeMemory-Bound Matrix-Vector Products GEMV Generates exactly one token at a time. The processor must sweep through all model weights and traverse the entire KV cache just to emit a single token. Arithmetic intensity collapses to ~1–2 FLOPs per byte. As context lengths extend into hundreds of thousands of tokens, decode transitions from being weight-bandwidth-bound to KV-cache-bandwidth-bound. Every architectural choice boils down to where weights and KV states live, how fast they move to ALUs, and how many watts get burned in transit. 2. The Incumbents & The Hyperscale Cloud Silicon NVIDIA: The Programmable Dominance NVIDIA's thesis remains rooted in general-purpose accelerated computing. Rather than locking down rigid fixed-function pipelines, NVIDIA continuously iterates the Streaming Multiprocessor SM while preserving the CUDA ecosystem: Asynchronous Tensor Pipelines: On Volta mma.sync , 32 threads executed in lockstep. On Hopper wgmma.mma async , 128-thread warp groups issued background matrix instructions. On Blackwell tcgen05.mma , a single thread issues descriptor-driven tensor commands directly into dedicated Tensor Memory TMEM , synchronized via hardware memory barriers mbarrier . Rubin advances this with 6th-gen Tensor Cores supporting native NVFP4 and FP6. GB200/GB300 NVL72 Rack: NVIDIA stitches 72 GPUs and 36 Grace CPUs into a single flat, cache-coherent 13.5 TB address space using 5,184 passive copper cables ~2 miles of cabling per rack . Using direct-attach copper over short reaches saves ~20 kW per rack compared to optical transceivers. Rubin NVL576 Kyber : Scales the coherent domain to 576 GPU dies in a liquid-cooled chassis, collapsing cross-rack transceivers directly into switch ASICs via Co-Packaged Optics CPO / Quantum-X Photonics . Weight-Stationary Systolic Arrays: TensorCores feature large 256×256 systolic arrays on Trillium v6e, Ironwood v7, and TPU v8 . Activations flow through pre-loaded weights, hardwiring data reuse into the silicon grid and eliminating register file power penalties per MAC. Static VLIW Scheduling: 322-bit VLIW bundles schedule every compute instruction statically. The XLA compiler plans all DMA transfers between HBM, CMEM, and VMEM ahead of execution. Optical Circuit Switching OCS & Boardfly: Google scales up using Inter-Core Interconnect ICI switched optically via Palomar 3D-MEMS Optical Circuit Switches. Robotic micro-mirrors steer physical light paths to reshape topologies per workload or route around faulty nodes. For TPU v8i Zebrafish , Google deployed Boardfly, a high-radix hierarchical topology designed specifically to slash all-to-all communication latency in Mixture-of-Experts MoE dispatch. AMD Instinct: 3D Stacking & Open Standards AMD has paired conservative Compute Unit microarchitectures with aggressive advanced packaging and high memory density: 3D Hybrid Bonding TSMC SoIC : Vertically stacks compute dies XCDs directly on top of base I/O dies containing 256 MB Infinity Cache. Capacity Advantage: Shipping 192 GB MI300X , 256 GB MI325X , and 288 GB MI355X HBM3e allowed full 405B-parameter models in FP8 to reside inside a standard 8-GPU baseboard. Helios Rack & UALink: AMD is scaling out with the Helios 72-GPU rack MI455X , leveraging UALink Ultra Accelerator Link for coherent scale-up and UEC Ultra Ethernet Consortium over Broadcom Tomahawk 6 switch silicon. Engineered by Annapurna Labs, Trainium integrates compile-time dataflow with AWS's Nitro cloud hypervisor: Silicon Collectives CC-Cores : While GPUs burn SM compute cycles executing collective communication kernels NCCL , Trainium incorporates dedicated CC-Cores that run all-reduce and all-gather operations directly across NeuronLink ports in hardware. NeuronSwitch & SRD: Trainium3 introduces NeuronSwitch-v1, an all-to-all fabric eliminating multi-hop penalties during MoE routing, backed by AWS's SRD Scalable Reliable Datagram protocol over Nitro EFA. 3. The Radical Extremes: SRAM-Only Latency Machines While GPUs and TPUs use tiered HBM + SRAM hierarchies, Cerebras and Groq eliminated external DRAM entirely to maximize single-user autoregressive decode speed: Cerebras WSE-3: Fabricates an entire 300 mm wafer as a single 46,225 mm² chip with 900,000 cores and 44 GB of on-wafer SRAM, pushing over 1,800 tokens/sec on Llama 3.1 8B. However, scaling to trillion-parameter frontier models requires clustering dozens of CS-3 chassis because SRAM bitcells have hit physical scaling walls. Groq LPU Language Processing Unit : A deterministic spatial processor with 230 MB SRAM per chip. Instructions, memory routing, and latency are locked down to the exact clock cycle at compile time. The Hybrid Co-Processor Model: At GTC 2026, NVIDIA announced the Groq 3 LPX, deploying 256 SRAM-only LPUs as latency accelerators alongside Rubin NVL72 racks—routing compute-dense Attention to HBM GPUs and latency-critical MoE/FFN layers to LPUs. 4. The New Frontier: OpenAI's Jalapeño ASIC Unveiled at Hot Chips 2026, Jalapeño is OpenAI's clean-sheet inference ASIC co-designed with Broadcom, manufactured on TSMC N3P/N3E nodes, and powered by Samsung HBM4 delivering 15.4 TB/s memory bandwidth at a conservative 700W TDP ≤ 550W sustained . Key Microarchitectural Decisions in Jalapeño 1. Out-of-Order L1 CoresDumps static VLIW. Handles dynamic prompt lengths and tensor shapes via smart hardware prefetching without compiler stalls. 2. Sliced Memory HierarchyCores pair directly with HBM4 partitions, skipping power-hungry global crossbars and complex GPU L2 caches. 3. Small-Shape MXFP Systolic ArrayWeight-stationary array optimized for smaller tile dimensions, eliminating underfill penalties on small batch sizes. 4. Curry Rack System Architecture16 Katsu CPU trays paired with 16 Vindaloo ASIC trays 128 XPUs/rack connected via 4.8 Tb/s copper backplanes to Chana switch trays. The Big Bet: Rejecting Prefill-Decode Disaggregation PDD A major trend in inference infrastructure is Prefill-Decode Disaggregation PDD —running prefill on compute-dense clusters and transferring the KV cache over network switches to memory-dense decode clusters. OpenAI explicitly rejected PDD for Jalapeño, opting for a unified homogeneous pool for three reasons: Traffic Volatility: The ratio of prefill tokens to decode tokens changes dynamically throughout the day. Disaggregated pools leave one cluster stranded and idle. KV Cache Locality: Transferring millions of KV cache tokens across network fabrics burns bandwidth and introduces latency spikes. Speculative Decoding: Running draft models and verification passes on the same silicon die and local copper fabric eliminates distributed network serialization overhead. 5. Software & The Erosion of the CUDA Moat NVIDIA's primary competitive moat has historically been software CUDA, cuBLAS, cuDNN, CUTLASS, TensorRT . Today, two forces are dismantling that barrier: 1. Gluon & Linear Layouts OpenAI programs Jalapeño using Gluon built on Triton , which formalizes hardware memory management via Linear Layouts—a mathematical framework for memory swizzling and layout conversions that makes non-NVIDIA silicon easy to program. 2. AI-Synthesized Assembly Kernels Rather than waiting years for engineers to hand-write assembly kernels, OpenAI used internal models Codex / GPT-Astra to synthesize production kernels directly. When benchmarking DeepSeek R1 on Jalapeño, Codex generated production Multi-Head Latent Attention MLA kernels autonomously, with AI-generated MoE blocks running 1.5× to 1.8× faster than human-tuned code. 6. Comprehensive Architectural Comparison Matrix Vendor / Chip Target Compute Unit Memory & Bandwidth Numerics Interconnect Software NVIDIA B200 / B300 Train / Inf SIMT SM + 5th-gen Tensor 192–288GB HBM3e 8.0 TB/s FP4, FP8, BF16 72 GPUs NVLink 5, 1.8 TB/s CUDA, CUTLASS NVIDIA Rubin VR200 Train / Inf SIMT SM + 6th-gen Tensor 288GB HBM4 ~13.0 TB/s NVFP4, FP6, FP8 144–576 dies NVLink 6, CPO CUDA, FlashAttn-4 Google TPU v8t / 8i 8t: Train / 8i: Inf 256×256 Systolic MXU 216–288GB HBM3e 8.8 TB/s MXFP4, FP8, BF16 1k–9.6k chips Boardfly, OCS JAX, XLA, Pallas AMD Instinct MI355X AI Cluster SIMD CU + Wavefront MFMA 288GB HBM3e 8.0 TB/s MXFP4, FP6, FP8 72 GPUs Helios UALink ROCm, Triton, vLLM AWS Trainium3 Trn3 Cloud Scale Systolic + GPSIMD Vector 144GB HBM3e 4.9 TB/s MXFP4, FP8, BF16 144 chips NeuronSwitch Neuron SDK, NKI Cerebras WSE-3 Low-Latency 900,000 Spatial Cores 44GB SRAM 21 PB/s FP16, BF16, INT8 Wafer Mesh / SwarmX cerebras.pytorch, CSL OpenAI Jalapeño Inference ASIC OoO L1 Cores + MXFP Systolic HBM4 15.4 TB/s MXFP4, FP8 128 XPUs/rack Copper to 2k OCS Gluon Triton , Codex 7. Three Trends Shaping the Next Era of AI Hardware 1. The True Economic Driver: Tokens Per Megawatt TPS/MW Datacenters are strictly bounded by power substation lead times and cooling capacity. Hardware evaluation is shifting from $/FLOP to total operating efficiency $/Token and Tokens/Joule . Chips that maintain high efficiency under realistic interactive concurrency will dominate datacenter economics. 2. Packaging: Direct Copper vs. Silicon Photonics Under 2 meters, direct-attach passive copper backplanes dominate NVL72, Jalapeño Vindaloo because they consume zero transceiver power. Beyond 2 meters, Co-Packaged Optics CPO and Optical Circuit Switches OCS are replacing traditional optical transceivers to avoid the conversion power penalty. 3. Precision Scaling: The Microscaling Standard MXFP4 / NVFP4 Sub-8-bit computation has coalesced around Open Compute Project OCP Microscaling standards. By grouping small 4-bit vectors with shared 8-bit scale factors, architectures achieve 2× memory bandwidth reduction without the numerical accuracy collapse of older uniform INT4 quantizations. Conclusion: The Four Camps of Modern AI Silicon The AI chip landscape has matured past a single winner-takes-all race: The Merchant Giants NVIDIA, AMD : Provide the programmable foundation, massive developer agility, and high-water-mark rack-scale interconnects. The Cloud Hyperscalers Google TPU, AWS Trainium : Leverage compile-time static pipelines to bypass merchant margins for their internal cloud fleets. The Latency Purists Cerebras, Groq : Exploit extreme SRAM bandwidth for instant, real-time agentic reasoning loops. The Vertical Frontier Labs OpenAI Jalapeño : Co-design silicon directly around proprietary models, using AI-synthesized assembly kernels to squeeze every drop of efficiency out of every watt.