ReLoRA

Researchers propose sMuon, a method that adapts the Muon optimizer for low-rank fine-tuning by approximating the orthogonalization step via linearization and least-squares, using only matmul operation…

Researchers have developed ReLoRA, a knowledge-reusing re-adaptation framework that enables fast recovery of task-specific Low-Rank Adaptation (LoRA) adapters after large language model (LLM) base-mod…