[Tutorial] Understanding and Implementing the Muon Optimizer

Update: Muon → Muon/Dion3 experiments at scale!

Since writing this tutorial, I’ve been investigating whether Muon and Dion3 retain their convergence advantages over Adam under practical training conditions, as part of my work for PyTorch Conference NA 2026.

The work grew into a debugging saga involving learning-rate scheduling, distributed optimizer state, and a silent tensor aliasing bug.

I’ve collected the presentation and my OLMo-core PR in one place:

PyTorch Con NA 2026 project materials:

If you’re interested in how orthogonalized optimizers behave beyond the algorithm itself, come take a look!