Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelines

A new survey reframes alignment tuning for large language models as a data pipeline design problem, decomposing data construction into three stages: response synthesis, preference evaluation, and preference instantiation. The researchers organize existing methods into a unified taxonomy, identifying recurring design trade-offs and failure modes that influence optimization signals. The work highlights open challenges including prompt-level alignment, agentic settings, and alignment under evolving objectives.

arXiv:2605.26442v1 Announce Type: new Abstract: Much of the alignment tuning literature is organized around optimization objectives, while the construction of alignment data is often treated implicitly. In this survey, we adopt a data centric perspective and reframe alignment tuning as a pipeline design problem. We decompose alignment data construction into three interacting stages, response synthesis, preference evaluation, and preference instantiation, and use this framework to organize existing alignment methods into a unified taxonomy. Through this lens, we identify recurring design trade-offs and failure modes observed across prior alignment methods, and distill a set of high level principles that clarify how pipeline design choices influence the resulting optimization signal. Finally, we outline open challenges for alignment data pipelines, including prompt-level alignment, agentic settings, and alignment under evolving objectives.