Recycler: Cost-Free Latent Reasoning via Hidden State Recycling

ByteDance Seed researchers introduced Recycler, an architecture that feeds both the token embedding and the previous token's hidden state through a lightweight adapter into the next decoding step, preserving cross-step continuous semantics with negligible decoding overhead. The work builds on the finding that a recurrent multitoken prediction (MTP) head with only about 2% of the backbone parameters achieves a loss comparable to a standalone model roughly 25% its size. ByteDance Seed reports that Recycler, with a full pretraining, SFT, continued pretraining and RL pipeline, outperforms models with 2–3× more parameters, with the largest gains in reasoning and mathematics and stronger results when integrated earlier rather than later.

2026-10-25 Recycler: Cost-Free Latent Reasoning via Hidden State Recycling ABSTRACT Standard autoregressive decoding links successive steps only through sampled discrete tokens, discarding the continuous semantics encoded in previous hidden states. In contrast, recurrent multitoken prediction MTP heads propagate both token embeddings and hidden states across steps. Remarkably, an MTP head with only ∼ 2% of the backbone parameters achieves a loss comparable to that of a standalone model ∼ 25% its size, underscoring the value of continuous hidden state feedback. Motivated by this observation, we introduce Recycler, a simple and effective architecture that feeds both embedding and the previous token’s hidden state through a lightweight adapter into the next step, preserving richer cross-step semantics with negligible decoding overhead. We develop a complete training and inference pipeline covering pretraining, SFT, continued pretraining and RL. Experiments demonstrate strong parameter efficiency, particularly in reasoning and mathematics. Although earlier integration yields larger gains, Recycler remains effective when introduced at later stages, outperforming models with 2–3× more parameters. AUTHORS ByteDance Seed VENUE arXiv