Towards Spec Learning: Inference-Time Alignment from Preference Pairs

Researchers introduced spec learning, a framework that uses a brief user instruction and a small set of preference judgments to compile natural-language specifications for LLMs. These specifications condition models at inference time without parameter updates, outperforming direct preference optimization (DPO) on specialized domains with dense preference signals. The resulting specifications are human-readable and transparent.

arXiv:2606.24004v1 Announce Type: new Abstract: Steering a large language model LLM toward a desired behavior typically relies on an iterative process of hand-crafting a prompt based on a careful inspection of the model's responses. This is an involved, brittle, and error-prone process. Preference-based fine-tuning is a more rigorous but often prohibitively expensive solution. We propose spec learning, a framework that relies on a brief user instruction and a small set of preference judgments. These are compiled into specifications in the form of natural-language prompts for an LLM. Specifications condition LLMs at inference time, and no parameter updates to the underlying models are required. We show that the responses generated based on the compiled specifications often outperform direct preference optimization DPO on datasets from specialized domains whose preference signal is dense. Unlike opaque weight updates, the resulting specifications are human-readable and double as interpretable and transparent written embodiments of the preference signal that produced them.