One Model, Two Paths: Balancing Single-Pass and Looped Performance in Language Models

A new arXiv paper (2610.33517v1) reports that jointly training a pretrained Transformer with weighted losses on both single-pass and looped outputs substantially reduces single-pass degradation compared to looped-only training while preserving looped gains over ordinary continued training. The authors find most of the benefit comes from adjusting the single-pass loss weight, with guidance from the frozen original model further improving the looped output, and that validation-selected weights keep single-pass quality within a predefined tolerance of ordinary continued training. The work addresses the trade-off created when a model reuses middle-layer weights for extra computation, since shared weights mean training for looped predictions can hurt single-pass quality.

arXiv:2610.33517v1 Announce Type: new Abstract: Repeating Transformer layers can improve language-model performance without adding parameters. A pretrained model can be adapted to support both single-pass and looped inference by reusing middle-layer weights for additional computation. Because both paths share weights, training to improve looped predictions can degrade single-pass quality. We study joint training strategies using weighted losses on both outputs, with guidance from the frozen original model. Joint supervision substantially reduces single-pass degradation compared to looped-only training, while preserving looped gains over ordinary continued training. Most of this benefit comes from adjusting the single-pass loss weight, with guidance from the frozen original model further improving the looped output. With validation-selected weights, our method preserves single-pass quality within a predefined tolerance of ordinary continued training while improving looped predictions.