Where to Place the Query? Unveiling and Mitigating Positional Bias in In-Context Learning for Diffusion LLMs via Decoding Dynamics

Researchers have identified and characterized positional bias in in-context learning for diffusion large language models (dLLMs), showing that query placement significantly affects generation quality. They propose Average Confidence, a metric to track decoding dynamics, and introduce Auto-ICL, a training-free adaptive routing strategy that optimizes query placement to robustly approach oracle performance across tasks.

arXiv:2606.19349v1 Announce Type: new Abstract: While In-Context Learning ICL is extensively studied in Autoregressive AR LLMs, its mechanism within Diffusion Large Language Models dLLMs remains largely unexplored. Unlike AR models restricted by unidirectional causal masking, dLLMs intrinsically utilize bidirectional attention, offering extensive spatial flexibility for query placement. Unfortunately, current practices conventionally inherit AR-style trailing-query templates, often overlooking the structural paradigm shift. This paper presents a comprehensive analysis unveiling that query position is actually a first-order variable in dLLMs. Through empirical decoupling, we demonstrate that positional variance impacts generation quality on par with example semantic quality. Internally, this positional sensitivity stems from a spatial Recency Effect'' in attention flow and task-dependent shifts in decoding trajectories. To mitigate this instability without ground-truth labels, we reveal that traditional single-step confidence $C {decoded}$ fails in dLLMs. Instead, we propose Average Confidence $\overline{C}$ , a novel metric tracking the iterative decoding process. By establishing the foundational spatial ICL baselines, we introduce Auto-ICL, a training-free adaptive routing strategy that dynamically optimizes query placement, robustly approaching oracle performance across heterogeneous reasoning and perception tasks.