[Tutorial] Understanding and Implementing the Muon Optimizer

A tutorial on the Muon optimizer has been updated with large-scale experiments testing whether Muon and Dion3 retain their convergence advantages over Adam under practical training conditions, work conducted for a PyTorch Conference North America 2026 presentation. The author reports the effort became a debugging saga involving learning-rate scheduling, distributed optimizer state, and a silent tensor aliasing bug, with the presentation and an OLMo-core pull request collected in one place.

Update: Muon → Muon/Dion3 experiments at scale Since writing this tutorial, I’ve been investigating whether Muon and Dion3 retain their convergence advantages over Adam under practical training conditions, as part of my work for PyTorch Conference https://events.linuxfoundation.org/pytorch-conference-north-america/program/schedule/?topic=training NA 2026. The work grew into a debugging saga involving learning-rate scheduling, distributed optimizer state, and a silent tensor aliasing bug. I’ve collected the presentation and my OLMo-core PR in one place: PyTorch Con NA 2026 project materials: If you’re interested in how orthogonalized optimizers behave beyond the algorithm itself, come take a look