arXiv:2611.22191v1 Announce Type: new Abstract: Block-wise diffusion language models (DLMs) refine multiple tokens in parallel, offering an alternative to token-by-token autoregressive decoding. Their long-context inference, however, remains bottlenecked by memory-bound attention: every denoising step repeatedly reads a growing historical key--value (KV) cache. Directly applying sparse attention does not fully solve this problem, because different queries can select different pages, inflating the block's collective read set, while repeated support construction adds overhead and permanent eviction can discard information needed later. We present RISA, a recoverable sparse-attention method with shared read plans for block-wise DLMs. RISA exploits the fact that historical KV remains fixed within a block while queries evolve across denoising steps. Newly generated history is organized into exact priority and recovery tiers; seed queries then build a protected, bounded read plan whose indices are reused across the block while attention is recomputed with fresh queries and current-block KV. This separates history preservation from repeated reading and amortizes sparse-support construction without changing the backbone or decoder. On DreamReasoner-8B, RISA retains comparable task performance. At 32K, the default configuration speeds up attention by $2.67\times$ and fixed-trajectory requests by $1.36\times$ over dense, and attention speedup reaches $5.53\times$ at 128K. Across eight settings on three models, RISA decodes faster than the tested MAGE, QUEST, and LoSA budgets at matched KL.
RISA: Recoverable Sparse Attention for Block-Diffusion Language Models
Researchers introduced RISA, a recoverable sparse-attention method for block-wise diffusion language models that reuses a shared, bounded read plan across denoising steps instead of rebuilding sparse support each step. On DreamReasoner-8B, RISA retained comparable task performance while speeding up attention 2.67x at 32K context and 1.36x for fixed-trajectory requests versus dense attention, with attention speedup reaching 5.53x at 128K. Across eight settings on three models, RISA decoded faster than the tested MAGE, QUEST and LoSA budgets at matched KL.
For block-diffusion long-context inference, build a shared sparse read plan from seed queries and reuse it across denoising steps; the source measured 2.67x attention speedup at 32K and 5.53x at 128K.
more in playbook: Inference and self-hostingRun your AI side-project on zahid.host
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.