{"slug": "approximate-muon-with-low-rank-adapters", "title": "Approximate Muon with low-rank adapters", "summary": "Researchers propose sMuon, a method that adapts the Muon optimizer for low-rank fine-tuning by approximating the orthogonalization step via linearization and least-squares, using only matmul operations. In experiments across supervised fine-tuning and ReLoRA pretraining, sMuon provides moderate performance improvements over alternatives, addressing the incompatibility of Muon with LoRA.", "body_md": "arXiv:2608.14492v1 Announce Type: new\nAbstract: The Muon optimizer shows clear benefits versus alternatives when pretraining neural networks. However, it is used less frequently for parameter-efficient fine-tuning (PEFT). One potential reason is that the most common PEFT method, LoRA, does not naturally combine with Muon since it is not mathematically possible to orthogonalize the weight update given by a low-rank parameterization. In this paper, we address this issue by approximating the solution to a relaxed Muon objective in the low-rank setting via linearization and then least-squares. We provide an efficient implementation that uses matmul operations only, as opposed to more complex linear algebra decomposition routines. Our method, sMuon (small Muon), performs favourably across SFT and a ReLoRA pretraining experiment. While results are model- and eval-dependent, we find overall that using Muon for low-rank fine-tuning provides moderate performance improvements.", "url": "https://wpnews.pro/news/approximate-muon-with-low-rank-adapters", "canonical_source": "https://www.machinebrief.com/news/approximate-muon-with-low-rank-adapters-hw0y", "published_at": "2026-08-17 04:00:00+00:00", "updated_at": "2026-08-17 06:12:16.146909+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence"], "entities": ["Muon", "LoRA", "sMuon", "ReLoRA"], "alternates": {"html": "https://wpnews.pro/news/approximate-muon-with-low-rank-adapters", "markdown": "https://wpnews.pro/news/approximate-muon-with-low-rank-adapters.md", "text": "https://wpnews.pro/news/approximate-muon-with-low-rank-adapters.txt", "jsonld": "https://wpnews.pro/news/approximate-muon-with-low-rank-adapters.jsonld"}}