{"slug": "microsofts-maia-200-ai-accelerator-at-hot-chips-2026", "title": "Microsoft’s Maia 200 AI Accelerator at Hot Chips 2026", "summary": "Microsoft unveiled the architecture of its second-generation Maia 200 AI accelerator at Hot Chips 2026, a 3nm chip with 140 billion transistors, 750W TDP, and 10,000 TFLOPS of FP4 performance, designed for deployment in Azure data centers. The chip uses a Software Defined Local Access (SDLA) dataflow architecture and is built on TSMC 3nm, featuring 6 HBM3e stacks with 7TB/s bandwidth and an 820mm² die. Microsoft emphasized the chip's deterministic dataflow, hierarchical memory, and custom networking to support scale-up to 6,000 chips across 128 racks.", "body_md": "The fourth AI accelerator presentation of Hot Chips 2026 comes from Microsoft, who like so many other hyperscalers has gone into the business of designing their own AI accelerator. Now in their second generation, Microsoft is preparing to deploy their Maia 200 accelerator inside their Azure data centers. The 3nm chip was announced early this year, while today’s presentation will be the deepest Microsoft has gone into the chip’s architecture and the many design tweaks they have made to improve its performance over the original Maia 100.\n\nThis article is being written live from the presentation, so please excuse any typos.\n\n## Microsoft’s Maia 200 Accelerator at Hot Chips 2026\n\nMaia is notable right off the bat for not being quite as aggressive of a design in terms of chip size. Though by no means a small chip at 140 billion transistors, Microsoft has paired the accelerator with just 6 stacks of HBM3e memory, and the total chip TDP is just 750 Watts. Though as these are designed to go into racks (just as with any other AI accelerator), the performance and power consumption still add up quickly.\n\nToday is going to the story of how Microsoft designed the Maia 200. The talk will go over the goals of the architecture, the SoC and its architecture, and how it comes together to form the larger system, with software layered in on top.\n\nInference has a very diverse workload between prefill and decode. Mixture of expert modeling also creates a diversity of workloads. These and more were the design target of the Maia 200.\n\nMaia 200 uses the Software Defined Local Acces (SLDA) dataflow architecture. Microsoft makes a distinction of architecture along two dimensions: data access and data movement. SDLA is defined by the fact that the dataflow is software-defined, while data access is completely local. This organization is similar to the Flynn’s classic taxonomy of processors.\n\nThe SDLA architecture affords a high degree of determinism. The dataflow is controlled in software, and thus is set at compile time which means it is known is advance. Meanwhile Microsoft is able to avoid a lot of cross-talk and heavy bandwidth/fabric needs by keeping data access within the compute elements.\n\nHere is an abstract look at the control and data path flow of the architecture. DMA and processing elements run the data path; each one has its own hardware queue of instructions. Explicit sync is achieved via semaphores.\n\nControl, I/O, and compute are all running independently, and being chained/synced via the aforementioned semaphores.\n\nMicrosoft’s fundamental computing block is a tile. Each tile contains a tile tensor unit (TTU) their matrix math engine, the tile vector processor (TVP) with SIMD engines, and then the tile control processor (TCP). Rounding out th etile is a TDMA block, and then an L1 cache block.\n\nMicrosoft employs a hierarchical memory structure, with L1 (tiler) and L2 (cluster) cache layers below the memory layer. Data is only moved between layers when it is needed to do so.\n\nAs for I/O, Microsoft uses a custom NIC and interconnect protocol as their network fabric. Reliability is a first-class design parameter here, with hardware-based recovery mechanisms. This custom design is achieved with minimal die space on Maia 200.\n\nEach SoC has 8 Ethernet lanes, split up into four network planes.\n\nMaia 200 also heavily emphasizes load balancing both inside a chip and across all of the chips in a cluster. Effective load balancing is fundamental to allowing Maia 200 to scale up to large domains.\n\nMaia 200 is built on TSMC 3nm, with a 750 Watt TDP. There is 7TB/second of HBM bandwidth, accomplished via 6 HBM stacks. In terms of compute throughput, the chip offers 10,000 TFLOPS of FP4 performance. The SoC die is 820mm2 altogether.\n\nMicrosoft uses a fully connected quad topology for their networking.\n\nThere is no scale-out networking for Maia 200. It is all scale-up networking. Microsoft’s slide demonstrates 128 racks with 6000 chips all together. Microsoft took a bet on unified Ethernet here, and it has paid off with operational simplicity. Each rack has tier-0 switches to connect the nodes within the racks, and then tier-1 switches to connect the racks themselves.\n\nShifting over to the software side of Microsoft’s ecosystem, they didn’t just co-design the chip and system, but they co-designed the entire stack right on through the software. The software at the core of this is the Microsoft Collective Communication Library (MCCL).\n\nThe SDLA design of the hardware requires that the kernels running on the hardware be tuned for the hardware for the best performance.\n\nGEMM workloads are one of the fundamental workloads run on Maia 200. Above, Microsoft’s presentation lays out how a batch GEMM workload is handled on the chip. A gather operation takes place over the central GNOC, while the weights are loaded from HBM, and finally the output is pinned to the L1 SRAM.\n\nOf course, Maia 200 chips aren’t meant to be used alone; they are meant to be used together in a scale-up domain. So here is a look at the distributed workflow for batch GEMMs.\n\nData reuse is also a very important element of execution on Maia 200 for efficiency purposes, as it reduces the memory traffic flowing across the trip. Block interleaving is used to get fully overlapped tensor/SIMD operations.\n\nMicrosoft uses an adaptive logical topology for collectives. A collective can complete with a single hop and no overhead.\n\nMaia also supports a ring-based topology for when large transfers are needed.\n\nNow here is a look at the performance of the Maia 200 in GEMM benchmarks. Points on the left are memory-bound, points on the right are compute-bound. Maia gets very close to its theoretical maximum (roofline) in these benchmarks.\n\nMeanwhile attention benchmarks show an effective peak of 1.65 PFLOPS.\n\nFinally, here is a look at collective benchmarking. Microsoft is able to reach almost 1.3TB/second in BF16 AllReduce (with the chart also showing how the different broadcast implementations affect performance), while All2All peaks at 655 GB/second.\n\nAnd that is a quick look at Maia 200. Microsoft has opted for a data-movement-centric architecture with SDLA, and paired this with a unified all-Ethernet networking structure to keep the networking simple. Software is the third pillar of Maia 200, with the classical software development house co-designing the hardware and software in concert to optimize the very explicit flow of data.", "url": "https://wpnews.pro/news/microsofts-maia-200-ai-accelerator-at-hot-chips-2026", "canonical_source": "https://www.servethehome.com/microsofts-maia-200-accelerator-at-hot-chips-2026/", "published_at": "2026-08-25 22:45:29+00:00", "updated_at": "2026-08-25 23:43:52.020950+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-chips", "artificial-intelligence"], "entities": ["Microsoft", "Maia 200", "Azure", "Hot Chips 2026", "TSMC", "HBM3e", "Maia 100"], "alternates": {"html": "https://wpnews.pro/news/microsofts-maia-200-ai-accelerator-at-hot-chips-2026", "markdown": "https://wpnews.pro/news/microsofts-maia-200-ai-accelerator-at-hot-chips-2026.md", "text": "https://wpnews.pro/news/microsofts-maia-200-ai-accelerator-at-hot-chips-2026.txt", "jsonld": "https://wpnews.pro/news/microsofts-maia-200-ai-accelerator-at-hot-chips-2026.jsonld"}}