OpenAI took the Hot Chips 2026 stage on Day 2 to detail Jalapeño, an in-house inference ASIC and system built with Broadcom and designed to be the best compute platform for OpenAI’s own inference workloads. Richard Ho, Ravi Narayanaswami, and Chris Leary walked through the chip’s roughly nine-month path from initial RTL to tapeout, its performance positioning against NVIDIA GB200 and GB300, and an architecture built around HBM4 and a spatial programming model.
We are doing this one live from the session, so please excuse typos.
OpenAI Jalapeno ASIC at Hot Chips 2026 #
OpenAI Jalapeño is framed as an inference platform rather than a raw accelerator. OpenAI is talking about this in terms of the silicon, together with its host and accelerator rack pair, targeting state-of-the-art performance per watt at low latency for multi-chip workloads, aided by AI-accelerated hardware and software co-design.
This project moved quickly once OpenAI concluded that inference and agentic workloads needed a purpose-built design. This timeline shows an architecture concept in late 2024, an RTL freeze in 2025, a late 2025 tapeout, Codex running in early 2026, with ChatGPT on the chip not long after.
OpenAI frames the design around two metrics, time to last token for user experience and tokens per joule for inference efficiency. Across those, it compares systems along the full Pareto frontier of request latency versus energy per token rather than chasing raw chip counts, throughput per chip, or time to first token.
For comparisons, OpenAI uses InferenceX, a public, power-normalized benchmark across a basket of open-source models that spans the full prefill-to-decode spectrum. Runs normalize to the package TDP, with Jalapeño at 700 watts against the GB200 at 1.2 kilowatts, and the GB300 and MI355X at 1.4 kilowatts, and OpenAI measured against the July 2026 Pareto frontier across other accelerators. Jalapeño uses single-token prediction, whereas the NVIDIA baselines it is compared against use multi-token prediction. This figure shows how MTP uses seven tiny draft-model turns plus one batched large-trunk pass to produce up to eight output tokens, reducing the number of expensive large-model passes by up to 8x.
OpenAI ran GPT-OSS to stress latency limits, DeepSeek R1 is a good draft-model case, and the 1-trillion-parameter Kimi K2.5 shows scaling across many devices. OpenAI notes none of the three were co-designed for Jalapeño and that it got all of them running between when A0 silicon returned to the lab and now.
Here is the throughput-per-kilowatt frontier for the GPT-OSS 120B model. Jalapeño sits on the Pareto frontier at 700 watts, compared to the GB200 at 1,200 watts, meaning it holds a lead in useful throughput per kilowatt at equal latency.
At matched operating points on GPT-OSS 120B, Jalapeño shows about 1.9x higher peak mixed tokens per second per kilowatt and roughly 1.7x lower end-to-end latency. At the previous best time between tokens, it delivers around 53.7x more throughput, which is the dramatic end of the gain.
This DeepSeek R1 670B MXFP4 run keeps the same story going at a larger scale. Jalapeño again holds the frontier at 700 watts against the GB300 at 1,400 watts.
On DeepSeek R1, the matched-operating-point gains widen. Jalapeño delivers about 1.7x higher peak mixed tokens per kilowatt per second and 3.6x lower end-to-end latency, and at the previous best TBT, it offers roughly 104.3x more throughput.
OpenAI also runs Jalapeño in single-token mode directly against the GB300 in multi-token mode on DeepSeek R1. Even without the speculative-decode advantage, Jalapeño pushes the throughput-per-kilowatt frontier forward.
At matched operating points in that direct comparison, single-token Jalapeño still leads the multi-token GB300, with about 1.5x higher peak mixed-token rate per kilowatt and 2.2x lower end-to-end latency. This margin narrows relative to the plain GB300 comparison but remains clearly positive.
This 1-trillion-parameter Kimi K2.5 run is the scaling test. Jalapeño again holds the frontier at 700 watts against the 1,400-watt GB300 on the MXFP4 deployment.
Kimi K2.5 shows the largest relative gap in the set. Jalapeño lands at about 1.5x higher peak mixed tokens per second per kilowatt and 3.4x lower end-to-end latency, with roughly 56.1x higher throughput than the previous best TBT.
OpenAI also claims sub-millisecond token-to-token latency on frontier models at economical throughputs and notes that multi-token prediction would add another 3x to 5x latency improvement at iso-efficiency. Hands are hurting at this point.
Moving into the architecture, OpenAI argues that a single request spans three hardware stages including a compute-bound prefill, a small draft model at ultra-low batch for latency-sensitive speculation, and a memory-bandwidth-bound spec-verify decode with bursty MoE communication. Each phase hits a different bottleneck, so efficiency only matters if the whole request stays within its end-to-end latency target.
Workload mixes shift the balance among prefill, draft, and verify, which OpenAI argues makes specialized, heterogeneous fleets inefficient, since idle accelerators still draw baseline power. Jalapeño answers with a single balanced chip, where the blocks that idle for one phase are gated, rather than paying for a separate accelerator’s baseline package, HBM, I/O, and network power.
This system-level design decision comes down to whether KV cache state moves between specialized systems or whether the active silicon mix changes within a single chip. OpenAI keeps KV local on Jalapeño and varies the active ratio of compute, memory, and network per phase, with unused units going dark rather than forcing KV to travel across a network at every phase boundary.
Jalapeño’s raw HBM4 bandwidth is striking on paper. At a 128-chip aggregate of more than one petabyte per second, a bandwidth-only ceiling implies 1,000 to 2,000 tokens per second per user without speculative decoding and 5,000 to 10,000 with it, yet OpenAI acknowledges the real system lands well below either rate, indicating that raw bandwidth is not the only limiting resource.
Long-latency paths, contended unified memory subsystems, and expensive global memory fences all leave compute units waiting on data in transit, thereby reducing utilization.
This architecture responds by pairing each core slice with an HBM slice to provide a fast local view and adding a specialized, low-latency collective network for common cross-core patterns. A more flexible general NoC remains for traffic that does not need the fast path, so common operands stay local, and data movement does not dominate execution.
A local Jalapeño domain reaches 128 ASICs with low core-to-core latency, and a half-flattened two-level Clos topology built around Broadcom Tomahawk6 switches spans a global domain of 2,048 chips, with higher bandwidth for tensor parallel and lower bandwidth for expert parallel.
On execution, OpenAI says the specification converges through the design loop rather than being fully known up front. It emphasizes a measure, verify, learn, change, and repeat cycle that shortens the feedback loop between workload simulation, RTL, quality-of-results analysis, and DV and physical design.
OpenAI used its own internal AI model and the XLS hardware description language to speed the flow. It said major changes landed up to the day of the RTL freeze, plus measured PPA wins over a human baseline, such as 56 percent on a BF16 multiply, back-to-back with a 10 percent smaller matrix unit area.
Jalapeño’s programming model is built to be simple for humans and easy for frontier AI to program. This hardware exposes local tensors, fast local paths, and explicit communication so an AI search can handle the mapping, placement, scheduling, and pipelining that would be onerous for a person.
Jalapeño runs a spatial architecture in which the Gluon framework programs each physical core as a thread block. Each core group’s tensor, SIMD, and scalar engines are organized around a fast local memory view, and specialized collectives coordinate work across cores, while tensor layout information explicitly captures physical placement.
Starting from functionally correct kernels, an internal AI system drives the implementations to high performance. Its optimized attention and MoE kernels run 1.5x to 1.8x faster than existing expert-written implementations, validated end-to-end on-chip.
Jalapeño delivers 13.4 PFLOP/s of mxfp4 by mxfp4 matrix compute, 15.4 TB/s of HBM4 bandwidth across 216 GiB, and a 700-watt package, scaling to 27 EFLOP/s and 432 TiB across a 2,048-chip system, with the network split at 600 GB/s for the local 128-ASIC domain and 200 GB/s for the global 2,048-ASIC domain.
OpenAI frames Jalapeño as Gen 1 of a multi-generation roadmap that compounds learnings over time. Gen 2 targets better performance per watt, and Gen 3 aims for economical, low-latency serving, as the architecture unlocks aggregate HBM bandwidth rather than simply adding more raw bandwidth.
They also give a shout-out to Broadcom and Celestica.
Final Words #
Building an in-house ASIC in about 9 months is wild. It really shows a good example of how AI is transforming industries. What happens when product cycles that use AI design and optimization are compressed? I actually found the software stack really interesting because OpenAI is building it for AI programming, which, of course, makes sense for them. Maybe the more tantalizing bit is how this is a proof point for other companies doing design. It is always neat to see more new hardware.