Supportive Token Revealing for Fast Diffusion Language Model Decoding

Researchers have developed AXON, a training-free module that improves the quality-latency trade-off in discrete diffusion language models by selecting which masked tokens to reveal based on their ability to support later denoising steps. Unlike existing methods that focus solely on avoiding unsafe token commitments, AXON monitors uncertain masked positions and intervenes when additional context is needed, shifting the criterion from safest reveals to those that best aid subsequent decoding. Experiments on reasoning and code-generation benchmarks show AXON reduces the number of function evaluations while maintaining or improving accuracy across multiple diffusion language models.

arXiv:2606.04236v1 Announce Type: new Abstract: Discrete diffusion language models can generate text efficiently by updating multiple masked positions in parallel, but this parallelism introduces a quality-latency trade-off. Aggressive decoding may commit mutually dependent tokens too early, while conservative decoding requires many denoising steps. Existing methods address this tension by deciding which tokens are safe to reveal using confidence or dependency criteria. However, avoiding unsafe commits does not necessarily make the remaining masked sequence easy to decode, since uncertain tokens may depend on masked tokens, creating a bottleneck for denoising steps. We propose AXON, a training-free module that can be added on top of existing parallel decoding strategies for diffusion language models. Rather than replacing the base decoder, AXON monitors the remaining uncertain masked tokens and intervenes only when their current state suggests that additional context is needed. It then shifts the criterion from which tokens are safest to reveal to which confident reveals would best support later denoising. AXON selects anchors, confident masked tokens that uncertain positions attend to, using attention, uncertainty, and confidence signals. Experiments on reasoning and code-generation benchmarks across multiple diffusion language models show that AXON improves the quality-latency trade-off of existing parallel decoders, often reducing the number of function evaluations while maintaining or improving accuracy.