# OpenAI's upcoming Jalapeño chip looks like it'll be an inference beast > Source: > Published: 2026-08-25 14:00:00+00:00 OpenAI offered its closest look yet at its spicy new Jalapeño AI accelerator at the annual Hot Chips semiconductor development conference at Stanford on Tuesday. The chips, first [teased](https://www.theregister.com/ai-and-ml/2026/06/24/openai-gets-chippy-with-broadcom/5261697) earlier this year, were developed in collaboration with Broadcom, and are the first in a series of custom silicon from OpenAI, designed (in part) by AI, for AI. Compared to contemporary GPU systems from Nvidia, OpenAI says the parts will deliver both higher throughput and lower latency when they start trickling out later this year and reach volume production in 2027. To be clear, Jalapeño won’t replace OpenAI’s long-time hardware partners, which also happen to be some of its most important investors. OpenAI still needs compute for training, and the highly programmable nature of GPUs means that OpenAI is likely to deploy on AMD and Nvidia first and then transition to in-house silicon later. It’s also worth noting that AMD’s MI455X and Nvidia’s Rubin GPUs, also expected to ramp production in early 2027, are very different kinds of chips optimized for a mix of training and inference, whereas OpenAI’s custom silicon only needs to excel at one job: inference. ### Memory bandwidth is king When it comes to inference, compute is key but memory bandwidth is king. And based on early benchmarks OpenAI shared with the press before its Hot Chips presentation Tuesday, the chip is shaping up to be an inference beast. Testing on SemiAnalysis’ InferenceX [benchmark suite](https://www.theregister.com/on-prem/2026/03/07/unpacking-the-deceptively-simple-science-of-tokenomics/5228181) — presumably this is an unofficial test — shows OpenAI’s Jalapeño-based systems delivering between 1.5x and 1.9x more “AI work” at peak throughput, and 1.7x to 3.6x lower end-to-end latency than the competition across GPT-OSS-120B, DeepSeek R1, and Kimi K2.5. If the latter two seem like weird models for OpenAI to be testing against, it's not that OpenAI plans to use these chips to run competitors' models, it’s just the models InferenceX uses. In any case, the test shows that Jalapeño isn’t some model-specific architecture designed for maximum performance at the expense of programmability… cough, cough [Taalas](https://www.theregister.com/systems/2026/08/06/amd-acquires-ai-chip-startup-taalas-to-boost-inference-performance-by-etching-models-into-silicon/5284344). Meanwhile, for ultra-low-latency inference, which has become the hot new segment for AI infrastructure providers, OpenAI says its chips are 2.1x to 4.1x faster. At a system level — we're starting here because the frontier models OpenAI trains rarely run on a single chip any more — each Jalapeño system with its 128 accelerators packs 1.7 exaFLOPS of 4-bit compute, 27.5 TB of HBM4 and just shy of 2 petabytes a second of memory bandwidth. By comparison, AMD and Nvidia’s latest rack systems are faster, delivering 1.46x to 2x more compute and up to 12 percent more memory on Helios, but just 85 percent the memory bandwidth of OpenAI’s rack. As of writing, OpenAI hasn’t shared system-level power consumption, but based on what we know about the accelerators we’d wager each rack will use between 40 and 60 percent of the power of competing GPU systems. As usual, take all of these claims with a grain of salt. The competitive systems in question are Nvidia’s GB200 NVL72 and GB300 NVL72 rack systems, which launched in 2024 and 2025 respectively. OpenAI’s point of comparison also excluded speculative decoding, which uses a small draft model to boost inference performance by predicting the outputs of another. While it can be argued this makes for a cleaner apples-to-apples comparison, when you're measuring inference performance, you’re not just comparing hardware but the platform as a whole. ### Joining the rack-scale revolution Jalapeño’s system design won’t surprise anyone. The chip is designed around a rack scale architecture similar to [Nvidia’s NVL72](https://www.theregister.com/on-prem/2024/03/21/a-closer-look-at-nvidias-120kw-dgx-gb200-nvl72-rack-system/912087) or [AMD Helios](https://www.theregister.com/systems/2026/07/23/amd-attacks-the-rack-with-helios-systems-that-rival-nvidias/5277246). Each rack is equipped with 128 Jalapeño accelerators on board, each of which, we're told, is capable of churning out 13.4 petaFLOPS at MXFP4. That compute is fed by 216 GB of HBM4 memory (we assume six 12-high stacks) good for 15.4 TB/s of memory bandwidth. OpenAI is expected to shed more light on the system and shipping during its Hot Chips presentation on Tuesday, but from what we gather, the chip also appears to feature a large SRAM cache. According to Richard Ho, VP of hardware at OpenAI, the chip was designed to minimize data movement and keep intermediate operations and things like key value caches, which track state, on the chip. "We designed Jalapeño to minimize data movement and communication delays. This means that model state, including the KV cache used while generating a response, can be explicitly placed and kept local while the system activates the right combination of compute, memory, and networking for each inference phase,” the company explained in a blog post. A large SRAM cache would be the most intuitive way to go about this, and it's what we’ve seen the likes of SambaNova, dMatrix, and others do in their own AI XPU designs. However, unlike Nvidia’s Groq LPX racks, Jalapeño isn’t a one-trick pony. Instead, we’re told, it’s optimized for both the compute-heavy prefill operations where prompts are processed, and the memory-bandwidth-intensive decode phase where output tokens are generated. “A system that excels at one phase can lose that advantage while waiting for data or moving model state between different resources,” the company explained. ### Designed with AI for AI Perhaps the least surprising detail in all of this is that OpenAI used AI to help design, architect, and optimize the chip for inference, cutting time from inception to tape out (the point at which a semiconductor design is finished and ready to be sent off to the foundry for fabrication) to just nine months. This included using its models to optimize its inference-serving engines and to write custom kernels as new models were introduced. But, as impressive as that might sound, AMD and Nvidia have been doing essentially the same thing internally for a while now. In fact, at Advancing AI last month, the House of Zen opened its [ROCm.AI](https://www.theregister.com/ai-and-ml/2026/07/24/amd-vibe-codes-its-way-past-the-cuda-moat-with-rocmai/5278580) offering to the public. Moreover, building a chip that can scale to hundreds of thousands of accelerators isn’t a trivial task, and AMD and Nvidia likely have the edge with regard to programmability, what with GPUs being general purpose accelerators. In any case, Jalapeño won't be the last chip we see from OpenAI, assuming of course the bubble doesn’t pop before its second-gen chip tapes out. ®