At a pre-Hot Chips media briefing this week, OpenAI’s VP of hardware, Richard Ho, revealed results of first benchmarks for its clean‑sheet ASIC designed specifically for large language models (LLMs). The company’s in‑house “Jalapeño” accelerator, developed with Broadcom and Celestica, targets both high throughput and low latency in the same architecture—while addressing what Ho framed as the real enemy in modern AI systems: data movement.
OpenAI announced the chip in June, stating it was built from the ground up for current and future LLMs across the industry, developed from design to production in nine months (which itself was accelerated by OpenAI’s models).
This week, the company revealed that the tests of the chip and the systems around it showed significant performance advances, with Jalapeño serving more AI work per unit of power, while also returning responses more quickly. “Jalapeño delivers both higher throughput and lower latency with one architecture, where existing hardware systems often have to make a tradeoff between the two,” OpenAI said in an official statement.
To quantify Jalapeño’s advantages, Ho said in his briefing that OpenAI leaned on InferenceX, a public benchmark from SemiAnalysis that measures the full serving path for AI requests. On three models—OpenAI’s own GPT-OSS-120B, as well as external models like Kimi K2.5 1T and DeepSeek R1 670B—OpenAI reports that the Jalapeño chip delivers:
- 1.5 to 1.9× more AI work per watt at peak throughput versus comparable systems listed on InferenceX.
- 1.7 to 3.6× lower end‑to‑end latency.
- For highly interactive workloads, 2.1 to 4.1× higher performance than comparable systems.
Ho also draws attention to a methodological nuance: OpenAI reports single‑token prediction (STP) performance, without speculative multi‑token prediction (MTP) decode, while many public numbers include that optimization.
“Our numbers are without speculative decode. Something called STP, meaning doing single token prediction, versus some of the numbers you’ll see out there are actually multiple token predictions, which in essence result in about a three to five times boost,” Ho said. “So, our single token prediction is actually as good or beats some of the multi token prediction of the other models of the other chips.”
When comparing architectures, he said this distinction matters. OpenAI suggests Jalapeño is competitive even before you turn on aggressive, potentially model‑sensitive inference optimizations.
Not just a repurposed GPU; it’s a general-purpose processor for AI workloads
During this week’s briefing, Richard Ho emphasized that the new chip is not a repurposed legacy GPU, but a ground-up AI accelerator tailored specifically for transformer workloads. Designed around tight memory-compute co-design, it anchors a multi-generation roadmap. He also reiterated, as in the June announcement, that they’d used AI to accelerate the development of the Jalapeño chip.
“We basically started with a blank sheet of paper,” Ho said. “We came in and we looked at the LLM models, where the bottlenecks were, where the inner loops were, and what was going on, and I think we might be the first like large scale chip design team to really do that from scratch without a legacy architecture, without a legacy programming model they had to support.”
The result is a general‑purpose accelerator for AI workloads, rather than something narrowly tuned to OpenAI’s own models. In the benchmarks, he had emphasized that Jalapeño ran both internal and external/open models, highlighting it as a flexible platform rather than a single‑model engine.
Data movement is a critical bottleneck
Ho added that the defining problem in LLM compute is not just peak TFLOPs, but how much data has to move, and how far.
“Data movement was super critical,” he said, and became the starting point for the architecture. “If you’re forced to move that data around to different cores, it just takes up a lot of time and energy. So, we designed an architecture which minimizes that. There is close affinity between HBM banks and cores, and so if you’re able to take advantage of that in your programming model, then you eke out a lot of benefits.”
This “affinity” between HBM banks and compute cores is a key design point. Rather than treating the memory system as a flat, uniform pool, Jalapeno binds memory regions closely to compute resources, with the software stack aware enough to exploit that locality.
That means model state, including the KV cache used while generating a response, is explicitly placed and kept local while the system activates the right combination of compute, memory, and networking for each inference phase. The network is integral to the architecture and allows the entire workload to remain within one connected system.
According to OpenAI, the result is a balanced, versatile accelerator designed to adapt to shifting model architectures. It excels across both prefill and decode phases and dynamically adjusts to the fluctuating demands characteristic of agentic workloads.
On using AI to build a chip that can be programmed by AI
The nine-month tape-out milestone cited by OpenAI was achieved with the use of AI to design the chip itself. AI-driven workflows accelerated implementation exploration, tightened design, measurement, and verification cycles, and enabled continuous iteration across model workloads.
In his briefing, Ho said that Jalapeño is an example of AI assisted hardware design. OpenAI used earlier generation models to assist with chip design and bring up, and is now using its latest models to accelerate software optimization and kernel tuning. According to Ho, AI contributed both to arithmetic circuit optimization and to packing more compute into the die.
He noted that the team was able to bring up three separate LLMs on Jalapeño in about two months—a timeline he cited as evidence of both the architecture’s programmability and the effectiveness of AI assisted workflows.
Getting to roofline performance without thermal throttling
In his briefing, Ho talked about balancing compute, memory bandwidth, and networking to get very close to the roofline performance without thermal throttling. He said thermals and power are clearly part of the balancing act, and the Jalapeño chip targets about 700 W per accelerator, which is aggressive but still within the envelope of modern data center cooling solutions.
Ho also stressed that the system is designed to avoid the kind of power‑driven throttling engineers have grown accustomed to in high‑end GPU deployments.
“It doesn’t thermally throttle. There’s almost no thermal throttling here because it is balanced. It’s coming in at about 700 watts, so it’s well within the range of most cooling solutions.”
Roadmap, deployment
Ho re-iterated that Jalapeño isn’t a one‑off project, and that it’s the first in a multi‑generation roadmap. He said Gen 2 is already in deep development and Gen 3 is “taking shape.” Each generation builds on what the team learns and aims to further advance both efficiency and speed.
On deployment, OpenAI plans to start small‑volume rollouts at the end of this year, with a more significant ramp in 2027. Jalapeño will coexist with Nvidia hardware and other devices within OpenAI’s fleet. In the near term, Ho expects Jalapeño to be particularly attractive for low‑latency inference, such as ultra‑responsive code generation and other interactive workloads, where the combination of reduced data movement, memory affinity, and high bandwidth can be fully exploited.
Ho also emphasized that external sales aren’t the focus for the Jalapeño chip. OpenAI’s own compute demand is growing fast enough that simply meeting internal needs will consume capacity for the foreseeable future.
This was also underscored by Sarah Friar, CFO of OpenAI, in a blog this week: she said that, while Microsoft’s compute and Nvidia’s chips were foundational to OpenAI’s growth, its portfolio of chips and providers it relies on also includes AWS, AMD, Broadcom, Cerebras, CoreWeave, Oracle, SB Energy and SoftBank.
“We actively manage this portfolio for both capability and economics. We use premium systems where capability matters most and optimize for efficiency where scale and cost matter more,” she said in the post. “Preserving credible choice across providers, hardware, and deployment models lets us direct demand toward the strongest performance per dollar, maintain pricing discipline as market conditions change, and move with the frontier as stronger technology emerges. Direct control adds leverage where tighter integration can improve the entire system. We partner where the ecosystem helps us move faster and build where co-design creates a meaningful advantage.”
This is a balanced way of saying, as with the broader hyperscaler fraternity, they all need so much AI compute to satisfy demand that, in the short to medium term, all options are open and required right now. Hence, they won’t be replacing the GPU incumbents just yet. But with everyone developing their own ASICs and custom chips, there may just come a point in time when the power/performance tradeoffs push one solution over the tipping point into being the preferred approach.
Read also:
[OpenAI Hardware Chief: ‘AI Scaling Laws Will Continue’](https://www.eetimes.com/openai-hardware-chief-ai-scaling-laws-will-continue/)
[NXP Expands Industrial Endpoint Access with MCU Topology Discovery](https://www.eetimes.com/nxp-expands-industrial-endpoint-access-with-mcu-topology-discovery/)
[Andes Condor Closure Came Amid Broader Cost-Cutting Effort](https://www.eetimes.com/andes-condor-closure-came-amid-broader-cost-cutting-effort/)