Affix Cache for Diffusion Large Language Models

Researchers introduced ACache, an affix-oriented cache reuse mechanism for Diffusion Large Language Models (DLLMs) that selectively recomputes key-value states of critical Anchor Tokens to maintain accuracy while reducing recompute latency by up to 55.7% and improving end-to-end throughput by up to 1.68x compared to full recomputation. Built on Fast-dLLM, ACache recovers accuracy loss from direct affix-cache reuse by recomputing around 20% of affix tokens, and a shared-prefix prototype on Nano-vLLM demonstrated the gains.

arXiv:2608.26140v1 Announce Type: new Abstract: Diffusion Large Language Models DLLMs enable non-autoregressive decoding and bidirectional context modeling, but efficient inference remains challenging. Unlike autoregressive systems, whose key-value KV cache can be reused for shared prefixes, DLLMs couple the KV states of shared context tokens with evolving generated tokens through bidirectional attention, making naive cache reuse stale while full recomputation is expensive. We present ACache, an affix-oriented cache reuse mechanism for shared text spans in DLLMs beyond prefixes. ACache identifies a small request-specific subset of critical affix tokens, called Anchor Tokens, by measuring their influence on masked generation tokens, and selectively recomputes the KV states of only these tokens while reusing the remaining affix cache. Built on Fast-dLLM, ACache recovers the accuracy loss caused by direct affix-cache reuse across different settings when recomputing around 20% of affix tokens. We also build a shared-prefix prototype on top of the Nano-vLLM engine, showing that ACache reduces recompute latency by up to 55.7% and improves end-to-end throughput by up to 1.68$\times$.