{"slug": "an-advanced-system-architecture-breakdown-of-openais-jalapeno-inference", "title": "An Advanced System Architecture Breakdown of OpenAI’s Jalapeno Inference Accelerator", "summary": "OpenAI presented the architecture of Jalapeno, its general-purpose AI inference accelerator, at the Hot Chips 2026 conference, detailing a design built for end-to-end optimization rather than peak \"show\" FLOPs. The chip pairs Broadcom's Tomahawk 6 switch for networking with speculative decoding via multi-token prediction, and OpenAI used Google's open-source XLS high-level synthesis tool with its frontier models for block-level PPA optimization and formal verification, which contributed to a shorter design timeline. The presentation frames Jalapeno as generalizable to frontier model workloads, not a drop-in GPU replacement.", "body_md": "At the end of the Hot Chips 2026 conference, OpenAI presented their chip architecture, Jalapeno. Jalapeno is a general purpose AI inference accelerator designed primarily for OpenAI’s workloads but generalizable to frontier model workloads.\n\nMost commentators will look at the headline and think OpenAI built a magic black box that replaces GPUs overnight. **This couldn’t be further from the truth.** There is a rich set of system architecture tradeoffs and design methodology improvements that went into the chip design.\n\nThis post is an **advanced** case study into Jalapeno based on their presentation deck and related material. I will cover the entire architecture, including **what motivated it,** **what tradeoffs and alternatives were likely considered by their leadership**, and **how AI added value:**\n\n#### Part 1: System Specification and Constraints\n\n- The Pareto Frontier - System Level Tradeoffs \n  - Latency of LLM Requests\n  - Inference Efficiency - Energy per Request\n- Architectural Observations \n  - Theoretical Memory Roofline is Higher than User Token Throughput\n  - Unpredictable Workload Characteristics between Prefill, Speculate, and Decode\n  - Balancing Large KV States on a Single Chip\n- Sources of Latency that affect User Experience \n  - Network Interconnect with SOTA Breakdown\n  - Compute Stalls Caused by Operands Arriving Late\n\n#### Part 2: How the Hardware Architecture was Co-designed to Handle LLM Workloads\n\n- Network Architecture with Broadcom’s Tomahawk 6 Switch \n  - Alternative Architectures (2-Tier Fat-Tree, 3D/4D Direct Torus / Mesh Network, and Dragonfly)\n- Speculative Decoding with Multi-Token Prediction\n- Spatial Chip Architecture\n- Chip Design Specifications\n\n#### Part 3: How AI Assisted Chip Design\n\n- 🔒Three Major Agentic Coding Paradigms to Assist in Chip Design \n  - 🔒LLM - RTL Pipeline\n  - 🔒LLM - High Level Synthesis Pipeline\n  - 🔒Agentic Flow\n  - 🔒Why Agentic Flows Aren’t in Leading-Edge Chip Design Yet\n- 🔒How Jalapeno used AI: High Level Synthesis with Google XLS \n  - 🔒Compared to “Waterfall” Chip Design Methodology\n\n#### Part 4: SW Model Tuning and Performance Results\n\n- 🔒Software Performance Tuning\n- 🔒Results on Frontier Models\n\nAfter writing this post, I wonder whether **AI should be best tailored to enhance the steps within the current “waterfall” chip design methodology or enable a more iterative methodology.** Throughout this design, OpenAI leveraged an open source tool Google XLS that aided in block level optimization for PPA and formal verification leveraging their frontier models. Google XLS had its benefits in contributing to a shorter timeline, but I think the iterative, cross-domain (and often messier) nature of work is perhaps the more significant factor.\n\nPrior to Hot Chips, I studied the architecture of data centers from the ground up by attended five technical conferences. You will see several of my deep dives scattered throughout this post that form the basis of various cross-domain effects discussed here.\n\nAfter DAC, I wrote a deep dive on how AI is currently employed in Chip Design. I recommend you read this and a primer on the fundamentals of AI accelerators.\n\nAs always, if you’re an expert and notice a mistake in this article, please reach out to me so I can have it promptly corrected.\n\n# Part 1: System Specification and Constraints\n\nOf all the chip architectures presented at Hot Chips, Jalapeno captures a lot of **complex tradeoffs** facing the system definition and co-design of AI accelerators. Rather than optimizing for peak “show” FLOPs, Jalapeno was designed with end-to-end optimization in mind while maintaining **balanced performance** across frontier model workloads. \n\n## The Pareto Frontier - System Level Tradeoffs\n\nArchitecting custom silicon requires navigating through tightly coupled Pareto frontiers. Pareto frontiers across three macro-level layers include:\n\n- **Intelligence vs Service Cost vs Latency.** At the API / model layer, more intelligent models utilizing higher parameters counts, reasoning trees, or MoE improve performance, but increase memory footprint and FLOP requirement per token.\n- **Throughput vs Interactivity.** At the systems / inference layer, there is a tradeoff between prefill (compute heavy) and decode (memory heavy) operations where batch sizes have an impact.\n- **Compute Density (FLOPs/mm2) vs Energy per Bit vs Reticle Limits** . At the physical silicon and packaging level, increasing on-chip compute density is bound by lithography reticle limits. This forces a shift to chiplets to separate specialized functions in appropriate process nodes and 2D interconnects to connect them.\n\nIn additional, there are a few other important system-level and microarchitectural pareto frontiers:\n\n- **Memory BW vs Capacity vs Energy per Bit.** This is the classic “Memory wall” tradeoff where on-chip SRAM has massive BW but terrible area density. HBM has huge capacity, but limited by thermals and shoreline limits. You need a mix of both.\n- **Switch Radix vs Reach.** In the scale-up interconnect, high-radix switches (such as Broadcom TH6) better “centralize” multiple GPUs by reducing network hops.**However, a high switch radix is only as good as the max reach the furthest GPU can connect to it.** Thus, copper SI limits and optical power overhead often bounds the max number of GPUs that can reasonably connect to high radix switches in the given network architecture.\n- **Programmability vs Silicon Efficiency.** In general, more efficient silicon for a specific task tends to be less programmable. GPUs are highly programmable while fixed function, highly efficient ASICs are less so.\n- **Model Accuracy vs Compute/Memory Density.** Most AI Accelerators push the precision down to FP8 and FP4 to increase the compute density and ease memory BW pressure. However, this decreases dynamic range and add more compiler complexity to ensure model accuracy is maintained.\n\nBasically, the most powerful model in the world can define what is possible, but needs effective HW-SW co-design to deploy it economically.\n\nOpenAI considered these pareto frontiers and narrowed it down to two important ones:\n\n- End-to-End Request Latency vs Mixed Throughput / kW.\n- Decode Speeds vs Mixed Throughput / kW.\n\nThroughout this post, I will touch upon design considerations in several other pareto frontiers as well.\n\n#### Latency of LLM Requests\n\nTraditional semiconductor vendors tend to architect accelerators by optimizing HW metrics such as peak theoretical FLOPs (or “show FLOPS”) and unconstrained batch throughput.\n\nOpenAI inverted this paradigm and began with what the end user cared about. OpenAI observed that user experience is largely covered by **inference latency** that is typically quantified in three ways:\n\n- **Time to First Token (TTFT)** - Prompt processing speed during the prefill phase\n- **Time per Output Token (TPOT)** - The speed of token generation during the decode phase\n- **Time to Last Token** - Total turnaround time of the whole request\n\nStarting from the end-user requirements of ChatGPT, real-time API clients, and internal agentic workflows, OpenAI reasoned backwards to establish strict SLAs—specifically targeting TPOT and end-to-end request latency - all while maintaining generality to other frontier models.\n\n#### Inference Efficiency - Energy per request\n\nAt the cluster level, inference efficiency effectively sets a constraint on the power system of the rack itself for the aggregate users expected to use it.\n\nInference energy competes with energy per request based on the **batch size**: \n\n- **Smaller batches** enable lower single user latency across fast interconnects, but wastes idle compute logic because memory channels stream 10’s of GBs of weights per token pass.\n- **Larger batches** maximizes energy efficiency to keep ALUs operating, but introduces queueing delays for single users.\n\nBoth tradeoffs much be balanced to enhance overall user experience while ensuring economic feasibility and grid usage. Depending on the service tier, OpenAI defines an end-to-end latency target through a service level agreement (SLA). After this is satisfied, compute efficiency is optimized (requests / s / W). Basically:\n\n## Architectural Observations\n\nOpenAI made a few architectural observations about comparable GPUs that influenced their chip and system architecture.\n\n#### Theoretical Memory Roofline is Higher Than User Token Throughput\n\nOpenAI’s calculation of the theoretical memory roofline highlights a massive gap between the theoretical memory BW and actual token generation speeds.\n\nOpenAI calculates a theoretical “max” token throughput of ~2000 full model weight reads / second IF HBM throughput was the solve physical constraint. They assume a 128 chip aggregate HBM4 BW of 1 PB/s with a ~1T parameter frontier model with FP4 values. This max limit can be pushed up to 5000 - 10000 tokens / s per user with speculative decoding.\n\nIn reality, single user token rates hover around 20 - 200 tokens / second. This is limited by various overheads in parallelism, such as inter-chip network latency, synchronization delays, and NoC routing contention. This overhead often causes underutilization of HBMs that sit idle between tokens.\n\nTo recover cluster economics, operators typically increase the batch size of concurrent requests to saturate available HBM bandwidth. While batching optimizes server throughput, it does nothing to speed up the individual user prompts. Users are completely unaware and often don’t care about the other requests that are batched alongside theirs.\n\nThis memory BW can be pushed even higher, but runs into severe physics constraints I fully describe in this article, causing memory makers like SK Hynix to look into optical interconnects.\n\n#### Unpredictable Workload Characteristics between Prefill, Speculate, and Decode\n\nVirtually all AI accelerators and surrounding systems face a tradeoff between **generality** and **specificity** to successfully handle varying degrees of workloads needed to run inference on an LLM request. It would be optimal to have accelerators fully utilized 100% of the time, but this is hard to achieve in practice.\n\nThe figure above shows the major LLM processing steps: prefill and decode. In between these is an optional third step: speculate.\n\n1. **Prefill** constructs the attention context by taking in the input prompt tokens, passes them all in parallel through a transformer model and and populating the key-value (KV) Cache in memory.\n  1. Prefill is compute heavy where the time-to-first-token (TTFT) matters.\n  2. Prefill stresses the compute that affects the PDN di/dt and thermals where throughput / kW matter\n \n  1. Prefill also stresses local SRAM and interconnect BW before committing data to memory\n\n1. **Speculate** is an optional step that runs a lightweight “draft” model to predict the next K (say 5-10) tokens. These K forward passes are fairly cheap and are ran a single verification forward pass across all candidate tokens simultaneously to verify results.\n  - Speculate is primarily used to bypass the memory BW wall of the decode step\n  - Speculate can take the form of single-token or multi-token decode\n  - Speculate stresses the optical / SerDes system because communication latency matters when data packets traverses stages.\n\n1. **Decode** autoregressively generates the response to the request one token at a time. To generate each output token, the model must read the model weight + accumulated KVcache to calculate attention and compute a single token.\n  1. Decode is largely a memory bound operation that stresses the HBM because it requires streaming fill target model weight with All-to-All (in MoE) for every pass.\n  2. Decode can be optimized with techniques such as Grouped-Query Attention (GQA) and speculative decoding that help circumvent memory BW\n  3. Decode also tends to make workloads bursty in MoE models\n\nThe problem is that most workloads ratios vary wildly, making it difficult it is to perfectly anticipate the compute needs for every request.\n\nYou can perhaps design highly specialized chip to handle each type of workloads and chain them all up. However, this could lead to compute inefficiency from idle accelerators.\n\nLikewise, you can implement all three computations on a single balanced chip where parts of it are potentially underutilized.\n\nRavi Narayanaswami observed that **it’s worse to have future unsupported features than built in underutilized features.** He mentioned how “opportunity cost is higher than marginal cost because it has a greater regret factor”. This means its worth the additional marginal cost up front to include features than may be underutilized in “dark silicon” in the short term.\n\nMy take on this is that its almost like how we don’t design rigid, highly efficient CPUs to handle fixed options to meet our needs now; we need to allow for additional **programmability** on general enough chips for now to handle workloads in the future. When designing new AI accelerators, I don’t think its sufficient to look at raw performance on current workloads for show; you have to consider the longer term uncertainty as well. \n\n#### Balancing Large KV States on a Single Chip\n\nThe size of HBM and LLM models imposes several challenges on KVCache to handle large context windows from increasingly large files passed on by users (10-100’s GB / request).\n\nOne approach is to build specialized chips for each type of workload. However, this entire KV cache still needs to be transported over the entire external network.\n\nOpenAI does not believe that this is the long term solution. It believes that **dark silicon is a lesser evil than idle specialized accelerators**. Separate accelerators add to additional HW complexity that takes up space for other useful functions. Inactive blocks on a single balanced chip silicon minimizes complexity and black boxes chip functionality to users. \n\nOpenAI implements a single chip solution with KVCache in close proximity to compute, memory, and the network. This way, data movement is **balanced** across all three components.\n\n## Sources of Latency That affect User Experience\n\nOpenAI is trying to cut down is **latency, or more specifically, tail latency, to enhance user experience**. \n\nIn LLM inference, data is primarily transported through global reduce operations such as all-reduce and reduce-scatter. These operations transmit data across multiple processors to keep parallel data notes in sync.\n\nTail latency cause a straggler effect across millions of computations where the worst case data packet can hold up a big computation where all other data packets arrive in time.\n\nOpenAI notes a few sources of latency within the signal path and at the SW level:\n\n- **Network Latency** - the signal path from point A to point B incurs various latency limits due to time-of-flight and HW complexity.\n- **Memory Systems** - two things affect memory latency:\n  - **HBM read/writes** have inherent latency that typically ranges from 30 - 50 nanoseconds due to signal traversal in the TSV stack, reading the DRAM cell itself, and complexity in the base die.\n  - **Unified, aggregated memory systems** tend to have highly contending paths due to NoC routing, crossbar intersections, and bank conflicts\n\n#### Long Latency Path in Interconnect\n\nIn general, any additional circuit complexity to handle high speed signal integrity nonidealities adds latency.\n\nSignals that travel through scale up/out network interconnects flow through switches that each add latency. Shown above are some ballpark SOTA numbers as of Feb 2026, and we see that the time-of-flight (or what OpenAI referred to as “HW speed of light”) is small relative to other sources.\n\nSeveral things can be done to minimize latency from all possible sources. Nikola Nedovic points out a few things during an ISSCC 2026 forum “Emerging Low-Latency Optical Connectivity”:\n\n- **Network Architecture level -** Increase the switch radix to decrease the # of network “hops” that packets take\n- **Rack Level -** Connect GPUs directly with massive copper backplanes rather than traditional PCIe controllers, retimers, and ethernet / InfiniBand switches\n- **Component level -** Optimize FEC that introduces 10-100’s ns of fixed delay\n- **Chip Level -** Move to low latency CPO links. This is the most widely talked approach attracting a lot of investment attention. NVIDIA is pursuing WDM to scale throughout without increasing data rate, and Marvell is targeting OMIB with GeSi modulators.\n\nIn my opinion, OpenAI did not utilize CPO because it is not really ready for real time deployment and looked to other means to optimize latency in the present term.\n\n#### Compute Stalls Caused by Operands arriving late \n\nAccumulated latencies in the network from these physical sources cause operands to arrive late in registers. In the classical computer architecture sense, operands arriving late leads to “stalling” because PE arrays require precise lockstep synchronization of data. This leads to compute units sitting idle that are waiting for results of intermediate operations.\n\nLatencies accumulate from all of the physical effects in the previous section. They also accumulate from **network congestion and contention** where two data packets try to transmit on the same physical resource, such as reading / writing from the same memory bank or output node. These are some reasons why **many consider the interconnect one of the critical bottlenecks of scaling AI compute, both from a fundamental physics and networking standpoint.** \n\n# Part 2: How the Hardware Architecture was Co-designed to Handle LLM Workloads\n\nThese pareto tradeoffs and architecture observations influence the HW system architecture. OpenAI used Broadcom’s Tomahawk 6 switch and Celestica for rack scale integration.\n\n## Network Architecture with Broadcom’s Tomahawk 6 Switch \n\nOpenAI uses a “half flattened” bounded 2-hop Clos topology for predictable latency. This network architecture was designed to strike a balance between handling **tensor parallelism** and **MoE parallelism** with 2 hops max for data packets.\n\nJalapeno was co-designed with Broadcom’s existing Tomahawk 6 (TH6), the world’s first 102.4-terabits-per-second Ethernet switch chip. This is a high-radix architecture supporting up to 512 ports at 200GbE (or 128 ports at 800GbE / 64 ports at 1.6TbE) on a single die.\n\nHere, the switches are arranged and connected in two domains:\n\n- **Scale-up / Local domain** - One TH6 connects 128 Jalapenos at 800GBps each (4x200G each). This high BW is sufficient for tensor parallelism.\n- **Global domain** - Eight TH connect 2048 Jalapenos across scale out domain.  This ensure predictable latency and BW sufficient for MoE parallelism.\n\nThis architecture ensures that any Jalapeno can connect with each other in two hops max through local and global TH6s.\n\nIn my opinion, they probably could have used a better switch for the scale-up domain. Broadcom’s Tomahawk 6 is a radix and BW monster intended for scale-out to reduce total network hops, but has a latency of 600 - 700ns due to the HW complexity. The Tomahawk Ultra has an ultra low fixed latency of 250ns, but half the total throughput at 51.2Tbps.\n\n#### Alternatives \n\nOther network architecture options they could have considered include:\n\n- **2-Tier Fat-Tree.** This uses two layers of spine and leaf switches that guarantees inter-rack packets will traverse with 3 hops (leaf - spine - leaf) across 2 tiers of switches.\n  - This is used by NVIDIA GB200 InfiniBand SuperPOD reference architectures\n  - This allows for **huge rack scale scalability** with**medium radix switches**\n  - OpenAI did not use this because 3 hops adds **tail-latency jitter** and higher power consumption, compared to the 2 hop, high radix topology\n\n- **3D/4D Direct Torus / Mesh Network.** These use direct, point-to-point links between neighboring compute nodes.\n  - This is used by Google TPU 8t\n  - This is cost effective for static, regular data movement patterns such as tensor parallelism\n  - Jalapeno doesn’t use toruses because toruses incur high average hop counts, causing very unpredictable link latency.\n\n- **Dragonfly Topology.** In a dragonfly topology, high-radix switches are arranged into densely connected local groups that act as single \"virtual switches,\" which are then fully connected to every other group across the system.\n  - This is used in Googles TPU8i\n  - The main benefit is that this saves on optical transceiver costs\n  - Jalapeno doesn’t use Dragonfly because it introduces non-uniform global paths. Relieving congestion requires non-minimal adaptive routing (such as UGAL) that adds to packet unpredictability.\n\n## Speculative Decoding with Multi-Token Prediction\n\nSpeculative decoding is almost like how humans guess the next words of the sentence.\n\nIn the decode phase of normal prefill-decode routines, autoregressive generation is sequential where each token is processed step-by-step with a full forward pass and memory access. This “one at a time” approach limits system efficiency due to higher latency and underutilized HW.\n\nSpeculative decoding is an optimization technique used to break through the memory wall by predicting and verifying multiple tokens simultaneously. Here, the “trunk” refers to all the layers in the typical transformer architecture with multi-head attention, FF, and add/norm. The inputs get passed through this “trunk” to generate the output tokens.\n\nThere are two major categories of speculative decoding:\n\n- **Single-Token Prediction -** 8 forward passes produce 8 different tokens\n- **Multi-Token Prediction -** 7 forward passes on a cheap graph model and one forward pass on a large model. This was selected to support the user experience, energy efficiency requirements, and allow for a higher token throughput.\n\nThough Jalapeno’s microarchitecture was explicitly engineered from the ground up to support MTP, OpenAI is currently running single token prediction workloads initially to prove out silicon.\n\n## Chip Architecture - Spatial architecture\n\nStandard GPUs use a temporal SIMT where massive warp schedulers coordinate execution of thousands of threads in lockstep.\n\nJalapeno uses a spatial architecture where independent core slices are tiled with dedicated HBM interfaces. These cores are connected to specialized collectives allowing direct core-to-core streaming. Each core operates with its own dedicated L1 memory view, running local tensor/SIMD/scalar units independently without waiting on a central warp controller to dispatch instructions. ,\n\nTo maintain spatial programmability with a familiar SIMT interface, OpenAI designed **Gluon** as their proprietary kernel programming language. Gluon builds on top of Triton, an open-source compiler, but enables more fine HW control. \n\nGluon is designed so hardware-software co-designers could write code using familiar GPU abstractions (thread blocks) without needing to hand-craft spatial dataflow graphs. Think of this like a “hybrid” approach of retaining the temporal SIMT abstraction most people know while programming for a spatial architecture which is normally a pain in the ass to program for.\n\n## Chip Design Specifications\n\nJalapeno is manufactured on TSMC 3nm to maximize low-precision inference throughput / W on end-to-end workloads.\n\nJalapeno pairs a compute die with six HBM4 stacks with 3 on each side. The memory system provides a total of 15.4 TB/s and 216 GiB. Jalapeno consumes 700W and achieves between 3-13 PFLOPS/s for fp4 / fp8 matrix multiplication.\n\n*(For comparison: B200 delivers 9 PFLOPs / s and B300 15 PFLOPs /s fp4 compared to NVIDIAs 1000W+ chip, but keep in mind Jalapeno is only for inference.)*\n\nThe 2,048-chip Jalapeño cluster is a full pod / datacenter row spanning roughly 28 to 32 racks. The system itself provides a total of 27EFlops/s, 32PB/s, 432 TiB.\n\nNote that most of the performance gains relate to GB can be attributed to HBM4.\n\n# Part 3: How AI assisted in Chip Design\n\n## Agentic Coding Methods\n\nOpen AI leverages AI-assisted **high level synthesis** with Google XLS. Let me explain what that is in the context of three major approaches for applying AI to chip design.", "url": "https://wpnews.pro/news/an-advanced-system-architecture-breakdown-of-openais-jalapeno-inference", "canonical_source": "https://www.siliconcodesign.com/p/an-advanced-system-architecture-breakdown", "published_at": "2026-09-01 16:51:42+00:00", "updated_at": "2026-09-12 16:45:32.718538+00:00", "lang": "en", "topics": ["ai-chips", "ai-infrastructure", "ai-research", "ai-products"], "entities": ["OpenAI", "Jalapeno", "Broadcom", "Tomahawk 6", "Google XLS", "Hot Chips 2026"], "alternates": {"html": "https://wpnews.pro/news/an-advanced-system-architecture-breakdown-of-openais-jalapeno-inference", "markdown": "https://wpnews.pro/news/an-advanced-system-architecture-breakdown-of-openais-jalapeno-inference.md", "text": "https://wpnews.pro/news/an-advanced-system-architecture-breakdown-of-openais-jalapeno-inference.txt", "jsonld": "https://wpnews.pro/news/an-advanced-system-architecture-breakdown-of-openais-jalapeno-inference.jsonld"}}