The LR Death Spiral 🧨 — Muon/Dion3 Debugging Saga

I was supposed to be running optimizer experiments for my upcoming PyTorch Conference talk.

Instead, my learning rate entered a death spiral.

I was testing AdamW, Muon, Dion, and the latest Dion3 implementation in OLMo-core—and Dion3 broke things in some very interesting ways.

Turns out the debugging trail involved distributed optimizer state, LR scheduling, and a particularly nasty PyTorch tensor aliasing trap.

A fun debugging experience nonetheless.

I wrote up the whole adventure here:

More optimizer experiments coming before PyTorch Con!