🎤 I gave a talk at PyTorch Conference NA 2026!

Independent researcher Jen Wei presented findings at PyTorch Conference NA 2026 on whether the Muon and Dion3 optimizers retain their convergence advantages over Adam under practical training conditions, covering mixed-precision training, checkpointing and distributed recovery. The work included hands-on integration of Dion3 into OLMo-core, training experiments, and documentation of two debugging episodes: a learning-rate scheduling failure in distributed optimizer state and a silent PyTorch tensor aliasing bug. Presentation materials and an OLMo-core pull request were published on Hugging Face.

Still letting this sink in a little. I went to PyTorch Conference NA last week and presented my investigation into a question that had been driving my independent research: Do Muon and Dion3 retain their convergence advantages over Adam under practical training conditions? The work took me into mixed-precision training, checkpointing, distributed recovery, and a debugging saga involving learning-rate scheduling, distributed optimizer state, and a silent PyTorch tensor aliasing bug. Along the way, I worked hands-on in OLMo-core, integrated Dion3, ran training experiments, and documented the journey. This project now includes: The research and experiments: investigating optimizer behavior under practical training conditions. The debugging saga: Part 1 — The LR Death Spiral https://medium.com/@jenwei0312/the-lr-death-spiral-escaping-pytorchs-memory-traps-in-distributed-optimizers-da1d724a557f The season finale: Part 2 — The Anatomy of a Silent Tensor Aliasing Bug https://medium.com/@jenwei0312/taming-orthogonal-optimizers-at-scale-part-2-the-anatomy-of-a-silent-tensor-aliasing-bug-147431a9c7b0 The project materials: Presentation + OLMo-core PR on Hugging Face https://huggingface.co/datasets/bird-of-paradise/muon-dion3-pytorchcon-2026 I’m proud that I took this investigation from independent experiments to a conference presentation, and grateful for the opportunity to share it with the PyTorch community. Here’s to more questions about how models learn, more experiments, and more things to debug.