{"slug": "the-current-state-of-ai-chips", "title": "The Current State of AI Chips", "summary": "NVIDIA and Google are racing to dominate the AI chip market, with NVIDIA's Blackwell and Rubin architectures and Google's TPU v8 and Ironwood v7 leading the shift toward energy-efficient 'tokens per watt' performance, as datacenter power limits become the primary constraint. The article details how these chips balance compute-bound prefill and memory-bound decode phases, with NVIDIA's GB200 NVL72 rack saving ~20 kW per rack via copper cabling and Google's TPUs using weight-stationary systolic arrays.", "body_md": "Hardware Architecture Deep Dive12 min read·August 2026\n\nThe Current State of AI Chips: Architectures, Silicon Wars, and the Shift to Watts-Per-Token\n\nIn 2018, Turing laureates John Hennessy and David Patterson predicted that the death of Dennard scaling would spark a “Cambrian explosion” in domain-specific computer architecture. Eight years later, that explosion has arrived—and the battleground is no longer just theoretical FLOPs, but datacenter thermal ceilings and Tokens per Joule.\n\nToday, dozens of distinct silicon architectures compete for AI workloads: programmable SIMT GPUs, compile-time systolic ASICs, wafer-scale SRAM engines, deterministic spatial LPUs, and clean-sheet custom silicon taped out directly by frontier AI labs.\n\nMeanwhile, the economic bottleneck has transformed. When a datacenter is capped at a strict 50-megawatt grid connection, raw chip FLOPs don't generate revenue—sustained Tokens Per Megawatt (TPS/MW) does. Here is the full breakdown of how modern AI silicon actually works, where the tradeoffs lie, and how the hardware landscape is stratifying.\n\n1. The Physics of the Workload: Two Conflicting Regimes\n\nEvery AI accelerator is an answer to a mathematical dilemma: modern generative models (transformers, mixture-of-experts, diffusion models) are sequences of massive matrix multiplications (GEMMs) interleaved with memory-bound vector operations and autoregressive generation.\n\nPhase 1: Prefill / TrainingCompute-Bound\n\nLarge GEMM Computations\n\nStacks entire batches of prompt tokens against weight matrices in parallel. Arithmetic intensity is high (hundreds of FLOPs per byte loaded). The hardware is limited purely by raw tensor core density.\n\nPhase 2: Autoregressive DecodeMemory-Bound\n\nMatrix-Vector Products (GEMV)\n\nGenerates exactly one token at a time. The processor must sweep through all model weights and traverse the entire KV cache just to emit a single token. Arithmetic intensity collapses to ~1–2 FLOPs per byte.\n\nAs context lengths extend into hundreds of thousands of tokens, decode transitions from being weight-bandwidth-bound to KV-cache-bandwidth-bound. Every architectural choice boils down to where weights and KV states live, how fast they move to ALUs, and how many watts get burned in transit.\n\n2. The Incumbents & The Hyperscale Cloud Silicon\n\nNVIDIA: The Programmable Dominance\n\nNVIDIA's thesis remains rooted in general-purpose accelerated computing. Rather than locking down rigid fixed-function pipelines, NVIDIA continuously iterates the Streaming Multiprocessor (SM) while preserving the CUDA ecosystem:\n\nAsynchronous Tensor Pipelines: On Volta (mma.sync), 32 threads executed in lockstep. On Hopper (wgmma.mma_async), 128-thread warp groups issued background matrix instructions. On Blackwell (tcgen05.mma), a single thread issues descriptor-driven tensor commands directly into dedicated Tensor Memory (TMEM), synchronized via hardware memory barriers (mbarrier). Rubin advances this with 6th-gen Tensor Cores supporting native NVFP4 and FP6.\n\nGB200/GB300 NVL72 Rack: NVIDIA stitches 72 GPUs and 36 Grace CPUs into a single flat, cache-coherent 13.5 TB address space using 5,184 passive copper cables (~2 miles of cabling per rack). Using direct-attach copper over short reaches saves ~20 kW per rack compared to optical transceivers.\n\nRubin NVL576 (Kyber): Scales the coherent domain to 576 GPU dies in a liquid-cooled chassis, collapsing cross-rack transceivers directly into switch ASICs via Co-Packaged Optics (CPO / Quantum-X Photonics).\n\nWeight-Stationary Systolic Arrays: TensorCores feature large 256×256 systolic arrays (on Trillium v6e, Ironwood v7, and TPU v8). Activations flow through pre-loaded weights, hardwiring data reuse into the silicon grid and eliminating register file power penalties per MAC.\n\nStatic VLIW Scheduling: 322-bit VLIW bundles schedule every compute instruction statically. The XLA compiler plans all DMA transfers between HBM, CMEM, and VMEM ahead of execution.\n\nOptical Circuit Switching (OCS) & Boardfly: Google scales up using Inter-Core Interconnect (ICI) switched optically via Palomar 3D-MEMS Optical Circuit Switches. Robotic micro-mirrors steer physical light paths to reshape topologies per workload or route around faulty nodes. For TPU v8i (Zebrafish), Google deployed Boardfly, a high-radix hierarchical topology designed specifically to slash all-to-all communication latency in Mixture-of-Experts (MoE) dispatch.\n\nAMD Instinct: 3D Stacking & Open Standards\n\nAMD has paired conservative Compute Unit microarchitectures with aggressive advanced packaging and high memory density:\n\n3D Hybrid Bonding (TSMC SoIC): Vertically stacks compute dies (XCDs) directly on top of base I/O dies containing 256 MB Infinity Cache.\n\nCapacity Advantage: Shipping 192 GB (MI300X), 256 GB (MI325X), and 288 GB (MI355X) HBM3e allowed full 405B-parameter models in FP8 to reside inside a standard 8-GPU baseboard.\n\nHelios Rack & UALink: AMD is scaling out with the Helios 72-GPU rack (MI455X), leveraging UALink (Ultra Accelerator Link) for coherent scale-up and UEC (Ultra Ethernet Consortium) over Broadcom Tomahawk 6 switch silicon.\n\nEngineered by Annapurna Labs, Trainium integrates compile-time dataflow with AWS's Nitro cloud hypervisor:\n\nSilicon Collectives (CC-Cores): While GPUs burn SM compute cycles executing collective communication kernels (NCCL), Trainium incorporates dedicated CC-Cores that run all-reduce and all-gather operations directly across NeuronLink ports in hardware.\n\nNeuronSwitch & SRD: Trainium3 introduces NeuronSwitch-v1, an all-to-all fabric eliminating multi-hop penalties during MoE routing, backed by AWS's SRD (Scalable Reliable Datagram) protocol over Nitro EFA.\n\n3. The Radical Extremes: SRAM-Only Latency Machines\n\nWhile GPUs and TPUs use tiered HBM + SRAM hierarchies, Cerebras and Groq eliminated external DRAM entirely to maximize single-user autoregressive decode speed:\n\nCerebras WSE-3: Fabricates an entire 300 mm wafer as a single 46,225 mm² chip with 900,000 cores and 44 GB of on-wafer SRAM, pushing over 1,800 tokens/sec on Llama 3.1 8B. However, scaling to trillion-parameter frontier models requires clustering dozens of CS-3 chassis because SRAM bitcells have hit physical scaling walls.\n\nGroq LPU (Language Processing Unit): A deterministic spatial processor with 230 MB SRAM per chip. Instructions, memory routing, and latency are locked down to the exact clock cycle at compile time.\n\nThe Hybrid Co-Processor Model: At GTC 2026, NVIDIA announced the Groq 3 LPX, deploying 256 SRAM-only LPUs as latency accelerators alongside Rubin NVL72 racks—routing compute-dense Attention to HBM GPUs and latency-critical MoE/FFN layers to LPUs.\n\n4. The New Frontier: OpenAI's Jalapeño ASIC\n\nUnveiled at Hot Chips 2026, Jalapeño is OpenAI's clean-sheet inference ASIC co-designed with Broadcom, manufactured on TSMC N3P/N3E nodes, and powered by Samsung HBM4 delivering 15.4 TB/s memory bandwidth at a conservative 700W TDP (≤ 550W sustained).\n\nKey Microarchitectural Decisions in Jalapeño\n\n1. Out-of-Order L1 CoresDumps static VLIW. Handles dynamic prompt lengths and tensor shapes via smart hardware prefetching without compiler stalls.\n\n2. Sliced Memory HierarchyCores pair directly with HBM4 partitions, skipping power-hungry global crossbars and complex GPU L2 caches.\n\n3. Small-Shape MXFP Systolic ArrayWeight-stationary array optimized for smaller tile dimensions, eliminating underfill penalties on small batch sizes.\n\n4. Curry Rack System Architecture16 Katsu CPU trays paired with 16 Vindaloo ASIC trays (128 XPUs/rack) connected via 4.8 Tb/s copper backplanes to Chana switch trays.\n\nThe Big Bet: Rejecting Prefill-Decode Disaggregation (PDD)\n\nA major trend in inference infrastructure is Prefill-Decode Disaggregation (PDD)—running prefill on compute-dense clusters and transferring the KV cache over network switches to memory-dense decode clusters.\n\nOpenAI explicitly rejected PDD for Jalapeño, opting for a unified homogeneous pool for three reasons:\n\nTraffic Volatility: The ratio of prefill tokens to decode tokens changes dynamically throughout the day. Disaggregated pools leave one cluster stranded and idle.\n\nKV Cache Locality: Transferring millions of KV cache tokens across network fabrics burns bandwidth and introduces latency spikes.\n\nSpeculative Decoding: Running draft models and verification passes on the same silicon die and local copper fabric eliminates distributed network serialization overhead.\n\n5. Software & The Erosion of the CUDA Moat\n\nNVIDIA's primary competitive moat has historically been software (CUDA, cuBLAS, cuDNN, CUTLASS, TensorRT). Today, two forces are dismantling that barrier:\n\n1. Gluon & Linear Layouts\n\nOpenAI programs Jalapeño using Gluon (built on Triton), which formalizes hardware memory management via Linear Layouts—a mathematical framework for memory swizzling and layout conversions that makes non-NVIDIA silicon easy to program.\n\n2. AI-Synthesized Assembly Kernels\n\nRather than waiting years for engineers to hand-write assembly kernels, OpenAI used internal models (Codex / GPT-Astra) to synthesize production kernels directly. When benchmarking DeepSeek R1 on Jalapeño, Codex generated production Multi-Head Latent Attention (MLA) kernels autonomously, with AI-generated MoE blocks running 1.5× to 1.8× faster than human-tuned code.\n\n6. Comprehensive Architectural Comparison Matrix\n\nVendor / Chip\n\nTarget\n\nCompute Unit\n\nMemory & Bandwidth\n\nNumerics\n\nInterconnect\n\nSoftware\n\nNVIDIA B200 / B300\n\nTrain / Inf\n\nSIMT SM + 5th-gen Tensor\n\n192–288GB HBM3e (8.0 TB/s)\n\nFP4, FP8, BF16\n\n72 GPUs (NVLink 5, 1.8 TB/s)\n\nCUDA, CUTLASS\n\nNVIDIA Rubin (VR200)\n\nTrain / Inf\n\nSIMT SM + 6th-gen Tensor\n\n288GB HBM4 (~13.0 TB/s)\n\nNVFP4, FP6, FP8\n\n144–576 dies (NVLink 6, CPO)\n\nCUDA, FlashAttn-4\n\nGoogle TPU v8t / 8i\n\n8t: Train / 8i: Inf\n\n256×256 Systolic MXU\n\n216–288GB HBM3e (8.8 TB/s)\n\nMXFP4, FP8, BF16\n\n1k–9.6k chips (Boardfly, OCS)\n\nJAX, XLA, Pallas\n\nAMD Instinct MI355X\n\nAI Cluster\n\nSIMD CU + Wavefront MFMA\n\n288GB HBM3e (8.0 TB/s)\n\nMXFP4, FP6, FP8\n\n72 GPUs (Helios UALink)\n\nROCm, Triton, vLLM\n\nAWS Trainium3 (Trn3)\n\nCloud Scale\n\nSystolic + GPSIMD Vector\n\n144GB HBM3e (4.9 TB/s)\n\nMXFP4, FP8, BF16\n\n144 chips (NeuronSwitch)\n\nNeuron SDK, NKI\n\nCerebras WSE-3\n\nLow-Latency\n\n900,000 Spatial Cores\n\n44GB SRAM (21 PB/s)\n\nFP16, BF16, INT8\n\nWafer Mesh / SwarmX\n\ncerebras.pytorch, CSL\n\nOpenAI Jalapeño\n\nInference ASIC\n\nOoO L1 Cores + MXFP Systolic\n\nHBM4 (15.4 TB/s)\n\nMXFP4, FP8\n\n128 XPUs/rack (Copper) to 2k (OCS)\n\nGluon (Triton), Codex\n\n7. Three Trends Shaping the Next Era of AI Hardware\n\n1. The True Economic Driver: Tokens Per Megawatt (TPS/MW)\n\nDatacenters are strictly bounded by power substation lead times and cooling capacity. Hardware evaluation is shifting from $/FLOP to total operating efficiency ($/Token and Tokens/Joule). Chips that maintain high efficiency under realistic interactive concurrency will dominate datacenter economics.\n\n2. Packaging: Direct Copper vs. Silicon Photonics\n\nUnder 2 meters, direct-attach passive copper backplanes dominate (NVL72, Jalapeño Vindaloo) because they consume zero transceiver power. Beyond 2 meters, Co-Packaged Optics (CPO) and Optical Circuit Switches (OCS) are replacing traditional optical transceivers to avoid the conversion power penalty.\n\n3. Precision Scaling: The Microscaling Standard (MXFP4 / NVFP4)\n\nSub-8-bit computation has coalesced around Open Compute Project (OCP) Microscaling standards. By grouping small 4-bit vectors with shared 8-bit scale factors, architectures achieve 2× memory bandwidth reduction without the numerical accuracy collapse of older uniform INT4 quantizations.\n\nConclusion: The Four Camps of Modern AI Silicon\n\nThe AI chip landscape has matured past a single winner-takes-all race:\n\nThe Merchant Giants (NVIDIA, AMD): Provide the programmable foundation, massive developer agility, and high-water-mark rack-scale interconnects.\n\nThe Cloud Hyperscalers (Google TPU, AWS Trainium): Leverage compile-time static pipelines to bypass merchant margins for their internal cloud fleets.\n\nThe Latency Purists (Cerebras, Groq): Exploit extreme SRAM bandwidth for instant, real-time agentic reasoning loops.\n\nThe Vertical Frontier Labs (OpenAI Jalapeño): Co-design silicon directly around proprietary models, using AI-synthesized assembly kernels to squeeze every drop of efficiency out of every watt.", "url": "https://wpnews.pro/news/the-current-state-of-ai-chips", "canonical_source": "https://openrating.io/blog/current-state-of-ai-chips", "published_at": "2026-08-27 10:30:03+00:00", "updated_at": "2026-08-27 10:49:02.208819+00:00", "lang": "en", "topics": ["ai-chips", "ai-infrastructure"], "entities": ["NVIDIA", "Google", "Blackwell", "Rubin", "TPU v8", "Ironwood v7", "GB200 NVL72", "Grace CPU"], "alternates": {"html": "https://wpnews.pro/news/the-current-state-of-ai-chips", "markdown": "https://wpnews.pro/news/the-current-state-of-ai-chips.md", "text": "https://wpnews.pro/news/the-current-state-of-ai-chips.txt", "jsonld": "https://wpnews.pro/news/the-current-state-of-ai-chips.jsonld"}}