2026-10-25 ABSTRACT
Standard autoregressive decoding links successive steps only through sampled discrete tokens, discarding the continuous semantics encoded in previous hidden states. In contrast, recurrent multitoken prediction (MTP) heads propagate both token embeddings and hidden states across steps. Remarkably, an MTP head with only ∼ 2% of the backbone parameters achieves a loss comparable to that of a standalone model ∼ 25% its size, underscoring the value of continuous hidden state feedback. Motivated by this observation, we introduce Recycler, a simple and effective architecture that feeds both embedding and the previous token’s hidden state through a lightweight adapter into the next step, preserving richer cross-step semantics with negligible decoding overhead. We develop a complete training and inference pipeline covering pretraining, SFT, continued pretraining and RL. Experiments demonstrate strong parameter efficiency, particularly in reasoning and mathematics. Although earlier integration yields larger gains, Recycler remains effective when introduced at later stages, outperforming models with 2–3× more parameters.
AUTHORS
ByteDance Seed
VENUE
arXiv