Simplifying the Modeling of Arbitrary Conditionals in Natural Language

Researchers propose AC-GPT, a modification to causal Transformers that enables evaluating and sampling from arbitrary conditionals in a single forward pass without degrading standard left-to-right performance. The method preserves the next-token prediction objective, allowing existing LLMs to be fine-tuned for arbitrary conditioning, and outperforms baselines on modeling such conditionals.

arXiv:2606.14943v1 Announce Type: new Abstract: Causal Transformers model sequences through an autoregressive factorization of the joint distribution, which enables efficient left-to-right decoding and conditional likelihood computation. However, they cannot tractably sample from or evaluate arbitrary conditionals -- e.g., a block of text conditioned on past and future tokens. Recent work aims to solve this problem through novel architectures, but they often lead to sub-optimal modeling of such conditionals and degraded generations. We propose Arbitrary Conditionals GPT AC-GPT which introduces a simple modification to standard causal Transformers to enable evaluating and sampling from arbitrary conditionals -- including past, future, and mixed contexts -- within a single forward pass. Unlike prior approaches, our method preserves the standard left-to-right ordering and next-token prediction objective essential for both strong performance and efficient training on natural language. Crucially, this compatibility allows existing LLMs to be fine-tuned for arbitrary conditioning. Our empirical results indicate that our method outperforms baselines on modeling arbitrary conditionals, without degrading standard left-to-right performance.