A paper proposes Muon-RT, a variant of the Muon optimizer for large language model self-distillation that reuses the exponential moving average (EMA) teacher's trajectory instead of storing a separate momentum buffer, eliminating that persistent training memory. The method targets the memory overhead Muon adds on top of EMA-teacher self-distillation, which already retains a history of student updates.
Self-distillation methods for large language models with an exponential moving average (EMA) teacher already retain a history of student updates. Muon nevertheless stores a separate momentum buffer, increasing persistent training memory. We propose Muon with Reused Teacher (Muon-RT), which reuses th
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.