Muon

The NanoGPT speedrun repository reports that a collaborative effort has trained a language model to 3.28 cross-entropy loss on the FineWeb validation set in under 75 seconds on 8 NVIDIA H100 GPUs, a d…

Researchers propose sMuon, a method that adapts the Muon optimizer for low-rank fine-tuning by approximating the orthogonalization step via linearization and least-squares, using only matmul operation…

A new study from researchers at arXiv finds that the Muon optimizer's speedup in reaching the grokking threshold on modular arithmetic comes from orthogonalization via the Newton-Schulz iteration, not…

Researchers at NVIDIA have enhanced preconditioned gradient methods like SOAP and Muon to overcome computational cost and numerical stability challenges in large-scale LLM pretraining, demonstrating t…

A new study on arXiv (2607.13246v1) finds that Muon, an optimizer that reshapes gradient updates through approximate orthogonalization and has outperformed Adam and AdamW in large language model train…

Researchers unveiled a latent world model using an equivariant encoder and predictor that achieves invariant predictions across orientations, with training loss symmetry ensuring consistent performanc…

Efficient Long-hOrizon (ELO) learning, a new learned optimizer, surpasses traditional models like AdamW and Muon across diverse tasks including language modeling and image classification, requiring le…

Researchers improved associative recall in recurrent neural networks by orthogonalizing the mLSTM memory matrix during reads, inspired by the Muon optimizer. In noisy associative recall tasks, the met…

Researchers introduce Depth-wise Gradient Augmentation, a new optimization paradigm that transforms layer-wise updates to exploit structured relationships across deep neural network layers. Their inst…

Researchers introduced Muon$^p$, a new optimizer that uses fractional spectral-power updates to interpolate between Muon and gradient descent, improving finetuning performance on billion-scale models.…

Vlad Feinberg published a guide on how to land a job at a frontier AI lab focused on pretraining, emphasizing kernel-level performance work as the most direct path into the labs. The guide recommends …

DeepSeek-V4's million-token context capability stems from a hybrid attention architecture that compresses context before KV storage, reducing cache pressure. Together's early bring-up on NVIDIA HGX B2…

DeepSeek-V4 shifts the focus from simply offering a million-token context window to making that window practically usable through efficient attention mechanisms. Its architecture employs hybrid compre…