Still letting this sink in a little. I went to PyTorch Conference NA last week and presented my investigation into a question that had been driving my independent research:
Do Muon and Dion3 retain their convergence advantages over Adam under practical training conditions?
The work took me into mixed-precision training, checkpointing, distributed recovery, and a debugging saga involving learning-rate scheduling, distributed optimizer state, and a silent PyTorch tensor aliasing bug.
Along the way, I worked hands-on in OLMo-core, integrated Dion3, ran training experiments, and documented the journey.
This project now includes:
The research and experiments: investigating optimizer behavior under practical training conditions.
The debugging saga: Part 1 — The LR Death Spiral The season finale: Part 2 — The Anatomy of a Silent Tensor Aliasing Bug
The project materials: Presentation + OLMo-core PR on Hugging Face I’m proud that I took this investigation from independent experiments to a conference presentation, and grateful for the opportunity to share it with the PyTorch community.
Here’s to more questions about how models learn, more experiments, and more things to debug.