We optimized Intel Gaudi2 to deliver 60% higher decode throughput than NVIDIA H100 on a long-context inference benchmark, at 9.4x better cost efficiency.
At Standard Kernel, our mission is to generate high-performance systems software to unlock peak performance across all kinds of AI hardware. In this post, we demonstrate optimizing Qwen3.5-27B inference on an Intel Gaudi2 accelerator. We increased cached-prefill throughput by 3.21x and decode throughput by 2.84x through software optimization alone. The optimized Gaudi2 achieved 8% higher cached-prefill throughput and 60% higher decode throughput than an NVIDIA H100 80GB PCIe, at just 17% of its hourly rental cost. This translates to 9.4x more output tokens per dollar.
We took an accelerator that initially appeared uncompetitive and made it outperform H100 on this workload without changing the underlying silicon. We believe the future of AI inference will be heterogeneous, with software playing a central role in unlocking the performance and economics of diverse hardware architectures.
Bad Software Makes Good Hardware Uncompetitive #
In 2019, Intel acquired Habana Labs for approximately $2 billion, hoping to challenge NVIDIA in the AI accelerator market. By all accounts, Habana’s Gaudi2 chips had the specs to compete with NVIDIA’s flagship H100, with more memory (96 GB vs. 80 GB) and comparable bandwidth (2.45 vs. 3.35 TB/s). Yet Gaudi saw limited adoption, and in January 2025, Intel canceled Falcon Shores, the chip that was meant to carry the Gaudi line forward.
One of the biggest challenges for alternative accelerators is software support. NVIDIA’s software ecosystem benefits from years of investment in compilers, kernels, inference engines, and model-specific optimizations, resulting in other hardware having a harder time keeping up. Even when the hardware supports the necessary operations, the available implementations may be slow or incomplete. Getting a model to run efficiently can require substantial engineering across the inference stack.
Comparing hardware based on benchmarks is hard as a result. Performance depends as much on the software stack as it does on the underlying silicon, and poor results can reflect a lack of software optimization rather than a fundamental hardware limitation. We wanted to find out how much more performance we could get out of Gaudi2.
We Made Gaudi2 Outperform H100 with AI #
We compared a single Gaudi2 with a single NVIDIA H100 80 GB PCIe on the following workload:
- Model: Qwen3.5-27B-FP8
- Concurrency: 16 requests
-
Input length: 70,000 tokens per request
-
Shared prefix: 95% This workload resembles applications that repeatedly process long, largely shared contexts, such as coding agents working with conversation histories or large codebases. We measured prefill and decode separately.
We used the following vLLM and plugin versions for the results above:
- **Gaudi2 (baseline):** vLLM v0.32.0 with the`vllm-gaudi` plugin dated October 26, 2026 (commit`6a3dd47c41` ).
- **Gaudi2 (optimized):** The baseline above with our custom changes.
- **H100:** vLLM 0.32.0.
Our optimizations improved Gaudi2’s cached-prefill throughput by 3.21x and decode throughput by 2.84x over the baseline. The optimized Gaudi2 exceeded the H100’s measured throughput by 7.92% for cached prefill and 60.35% for decode.
The gains come from optimizing the entire software stack. Some of our optimizations are summarized below:
- Reduce prefill padding: use finer 128-token buckets instead of larger padded blocks. This reduces wasted computation and allows more requests to fit within the scheduling budget without splitting them.
- Speed up prefill attention: process cached prefixes separately from causal suffixes to avoid unnecessary masking. Use batched matrix operations, larger cache-aligned K/V tiles, and more efficient softmax normalization.
- Speed up GDN prefill: replace expensive 128×128 matrix inversions with smaller blocked inversions, accelerated by a custom TPC kernel. Reuse cached tensors and eliminate redundant forward passes.
- Reduce GDN computation and memory traffic: use FP16 for intermediate tensors while keeping numerically sensitive operations in FP32. Normalize Q/K before head expansion and batch independent matrix operations.
- Optimize normalization and convolution: replace manual RMSNorm reductions and causal convolution with faster implementations, preserving the original numerical behavior and state handling.
- Reuse shared K/V during decode: gather shared prefix data once instead of separately for each request, and maintain packed buffers for new tokens to avoid repeated data movement.
- Reduce GDN decode state traffic: keep recurrent state in dense buffers across decode steps instead of repeatedly reading and writing the cache. Write back only when necessary.
- Fuse decode operations: combine convolution, activation, Q/K/V extraction, and normalization into a single TPC kernel. Preserve recurrent state correctly across graph replay.
Better Software Makes Gaudi2 10x Cheaper per Output Token #
Gaudi2’s performance gains become even more interesting when we account for hardware costs. We found Gaudi2 available to rent for $0.60/hour, compared with $3.50/hour for H100. That’s nearly a 6x difference in hourly cost. On our benchmark, the optimized Gaudi2 not only outperforms H100 in throughput, but does so at a fraction of the hourly rental cost.
At these prices, optimized Gaudi2 delivers 9.4x more decoded tokens per dollar and 6.3x more newly computed cached-prefill tokens per dollar than H100. This illustrates how dramatically software optimization can change the economics of AI hardware. An accelerator that initially appeared uncompetitive can become a compelling alternative for certain production workloads, without any changes to the underlying silicon.
Making Hardware Efficient Shouldn’t Require Monumental Engineering Effort #
There are more AI accelerator architectures than ever, but adopting a new one remains a major engineering undertaking. Even determining whether a particular workload can run well on a chip often requires substantial porting and optimization work. As models evolve, much of that work needs to be repeated.
We’re building Standard Kernel to change this. Our optimization system can autonomously experiment with changes across the inference stack, benchmark them on real hardware, verify correctness, and accumulate what it learns, ultimately resulting in large speedups with little effort. Over time, we want to make optimizing for a new architecture a routine process rather than a major engineering project.
The implications extend beyond making individual models faster. If workloads can be ported and optimized across architectures with much less manual effort, customers have far more freedom in choosing which hardware to use, and chip manufacturers have a much easier time getting their hardware adopted. Not every accelerator will be the best choice for every workload, but it will be much easier to find out what each one is good at.
We’re continuing our work on Gaudi and other hardware architectures. If you’re building AI hardware, operating inference infrastructure, or looking to get more tokens per dollar, we’d love to work with you. We’re also hiring people who are excited by these problems. Get in touch at hi@standardkernel.com.