MUON-RT Reusing the Teacher's Trajectory for Momentum-Free Muon for LLM Self-Distillation

A paper proposes Muon-RT, a variant of the Muon optimizer for large language model self-distillation that reuses the exponential moving average (EMA) teacher's trajectory instead of storing a separate momentum buffer, eliminating that persistent training memory. The method targets the memory overhead Muon adds on top of EMA-teacher self-distillation, which already retains a history of student updates.

Self-distillation methods for large language models with an exponential moving average EMA teacher already retain a history of student updates. Muon nevertheless stores a separate momentum buffer, increasing persistent training memory. We propose Muon with Reused Teacher Muon-RT , which reuses th