Training Foundation Models with a Single Algorithm

Researchers report that the staged pipeline used to train foundation models — pretraining, midtraining, supervised fine-tuning (SFT) and reinforcement learning (RL) — is not algorithmically fundamental, and that suitable RL objectives can be derived as maximum-likelihood objectives so a single algorithm covers the whole process. The work argues the separation between these stages, which also involves changes in training data, can be collapsed into one method.

Foundation models are typically trained in stages: pretraining, midtraining, supervised fine-tuning SFT , and reinforcement learning RL , with changes in both data and learning algorithms. We show that this algorithmic separation is not fundamental. By deriving suitable RL objectives as maximum-li