# Improving Decode Throughput on Intel Gaudi 3 > Source: > Published: 2026-09-01 00:00:00+00:00 The Intel Gaudi 3 is a high-performance AI accelerator with impressive compute specs. Unfortunately, we found that for some model architectures, vLLM-Gaudi’s PagedAttention implementation significantly underutilizes the hardware, specifically when performing sliding window attention. Here, we discuss optimizations to serve Gemma 4 31B on Gaudi. First, we noticed that vLLM-Gaudi overallocates KV cache to sliding window layers. Patching this expanded the usable KV space by 3.5× (for 4k-context requests), allowing us to achieve higher batch sizes. This exposed an expensive KV gather operation that limited decode performance at greater concurrency. We changed how sliding window KV cache was allocated using a ring buffer, which raised decode MBU from 37% to 59% and decode throughput by 1.6×. ## Patching sliding window KV allocation Gemma 4 31B has 50 sliding window layers and 10 global attention layers. Global attention attends over the full context length, so the required KV for these layers grows linearly with sequence length. Sliding layers, however, only operate over the last 1024 tokens, meaning that we only need to store KV for that window. Stock vLLM-Gaudi does not take advantage of this feature and allocates the same full-context KV for all model layers. This waste limits the achievable batch size because we exhaust KV capacity too early. We fixed this by freeing sliding window KV when it is no longer within the 1024-token window, allowing it to be used for future requests. This patch fixed storage efficiency, letting us serve Gemma at reasonable batch sizes, but didn't change the efficiency of the underlying computation. At high batch sizes, the decode step was significantly slower than we expected. ## Expensive KV gathers Here’s a trace of sliding window attention using vLLM-Gaudi’s PagedAttention implementation at a batch size of 56: Because of a time-consuming KV gather operation, this implementation moves 1.07 GB of data over 1.10 ms. Compared to Gaudi’s peak of 3.7 TB/s, this is an MBU of only 29.5%. Much of this inefficiency comes from how KV paging is implemented on Gaudi. Paged KV is standard in modern LLM serving engines because it allows them to admit mixed-length requests on the fly, without needing a contiguous buffer allocated for the maximum possible context length. Instead, paging splits the KV cache into fixed-size pages that are allocated to live requests on demand (see [PagedAttention](https://arxiv.org/abs/2309.06180)). Unfortunately, the pointer indirection and non-contiguous memory access created by the technique do not map well to Gaudi hardware. The Gaudi 3 has 8 Matrix Multiplication Engines (MMEs) and 64 Tensor Processor Cores (TPCs), handling GEMMs and vector operations, respectively. The MMEs can’t collect scattered pages in memory by themselves. To perform attention over a set of scattered KV pages, the TPCs must first *gather* the selected pages into a preallocated workspace that the MMEs can read from. This gather is purely a rearrangement of data in HBM, so it contributes no useful model work. ### Contiguous PagedAttention vLLM-Gaudi introduced Contiguous PagedAttention to try to solve this challenge. It targets the gather step of the stock route by keeping KV pages contiguous in memory, allowing it to feed the attention kernel directly. To accomplish this, it must occasionally rearrange KV pages in HBM to be contiguous. Unfortunately, we can’t realize this gain when serving models like Gemma under reasonable workloads, because contiguous PA assumes that attention is always global. For sliding layers, we only need to cache KV and compute attention for the most recent 1024 tokens, but instead, it stores the full context and masks the majority when computing attention. Contiguous PA is built on the assumption of append-only cache buffers, so a constantly moving sliding window cache doesn’t map cleanly. At a context length of 4k tokens, the extra traffic and computation make its performance worse than the stock route. ## Our approach At this point, disabling KV paging entirely looks attractive. It would allow us to remove the gathers without data rearrangement. Unfortunately, it would create a large amount of cache waste, because we can’t know how much cache a request may need before it finishes decoding. Thankfully, for many models, we don’t need to keep the entire KV cache unpaged to capture a huge portion of the benefit. For Gemma, 50 of the 60 layers operate over just the most recent 1024 tokens. If we keep the KV for these layers cached contiguously, we can capture much of the benefit of a completely unpaged cache. To avoid having to shift the entire buffer for each token, we use a ring buffer. When a new request is admitted, we allocate contiguous K and V buffers. Importantly, we do this within the same HBM that backs global attention KV pages, using it as a shared pool. This allows us to balance ring buffers and global attention pages to match the shape of incoming requests. Short-context requests will use more space for ring buffers than for global attention pages. Conversely, a few long-context requests will have the opposite balance. For each decoded token, the new token’s KV is written and the head of the buffer is advanced by one position. Conveniently, after applying positional embedding to K, attention is invariant to a common reordering of K and V. Since the K and V ring buffers are written to in lockstep, they’ll always have the same cyclic offset. This means we never have to undo their rotation and can pass both buffers directly into a fused attention kernel! For sliding layers, our optimization removes the gathers entirely. This optimization shortens sliding attention by 2.5× and raises its MBU to 74.8%. ## Results At a decode batch size of 1, this improvement increases throughput by only 11%. However, as the batch size increases, the stock engine spends an increasing proportion of time gathering KV. While much of the model scales well with higher batch sizes, gathers are bandwidth-bound, so they increasingly dominate the time of a decode step. At a batch size of 56, our optimizations raise decode MBU from 37% to 59%, improving both interactivity and throughput by 60% over the patched baseline. Sail’s efficient inference runs primarily in high-batch regimes like this. ## What’s next Sail is always looking for the most efficient compute, and Gaudi has the raw specifications to be exceptional at LLM inference. We’ll continue to push the efficiency of modern LLM architectures like MoE, sparse attention, and linear attention on Gaudi. If you like learning about cool new chips and pushing performance to the roof(line), we’d love to [hear from you](https://www.sailresearch.com/careers)!