cd /news/ai-research/optimize-the-cob-then-the-kernel-opt… · home › topics › ai-research › article
[ARTICLE · art-172335] src=arxiv.org ↗ pub= topic=ai-research verified=true sentiment=↑ positive

Optimize the Cob, Then the Kernel: Optimizing Beam Search for Generative Recommendation Inference

A team of researchers reports in arXiv paper 2611.23339v1 that optimizing the generation pipeline holistically, without custom kernels, cuts end-to-end latency for generative recommendation inference by up to 25x versus vLLM on Ascend NPUs. The approach concatenates beams along the sequence dimension using Tree Attention to eliminate redundant KV cache accesses over the shared prompt prefix, and adds pre-generated partial-attention masks, static KV cache allocation, loop-unrolled generation dispatch, TorchAir graph compilation, automatic operator fusion, and vocabulary-constrained decoding. The authors argue the generation pipeline should be optimized holistically before resorting to custom kernel development, noting their generation latency is comparable to a state-of-the-art solution that uses custom kernels.

recommendation

For beam-search generative recommendation on Ascend NPUs, batch beams via Tree Attention with pre-generated partial-attention masks and static KV cache before writing custom kernels; the source measured up to 25x lower latency than vLLM.

more in playbook: Inference and self-hosting
by read1 min views7 publishedNov 19, 2026

arXiv:2611.23339v1 Announce Type: new Abstract: Generative recommendation (GR) inference differs substantially from conventional LLM serving, featuring long input sequences, wide beam search, and only a few fixed output tokens. Existing LLM serving engines such as vLLM process beams as batched independent sequences, causing redundant KV cache accesses over the shared prompt prefix. Furthermore, vLLM executes each beam search generation step as a separate engine invocation, incurring host-side beam processing overhead between decoding steps and substantial runtime KV block management in PagedAttention. Using Tree Attention, we concatenate beams along the sequence dimension to eliminate redundant KV accesses, and further optimize execution through pre-generated partial-attention masks, static KV cache allocation, loop-unrolled generation dispatch, TorchAir graph compilation, automatic operator fusion, and vocabulary-constrained decoding. Without the use of custom kernels, our approach reduces end-to-end latency by up to 25x over vLLM on Ascend NPUs, and achieves generation latency comparable to a SOTA solution using custom kernels, showing that the generation pipeline should be optimized holistically before resorting to custom kernel development.

── more in #ai-research 4 stories · sorted by recency
── more on @vllm 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/optimize-the-cob-the…] indexed:0 read:1min 2026-11-19 · —