A new survey of neural-network optimizers from arXiv (arXiv:2608.28557v1) finds that matrix-aware methods such as Muon, Shampoo, and SOAP represent a genuine advance, but no context-independent replac…
Researchers propose a curvature-conditioned multiscale momentum method with sphere constraints that accelerates LLM pretraining, improving Muon across dense and MoE architectures from 0.12B to 2.3B pa…
Shrijith Venkatramana, an engineer building LiveReview, explains how the Adam and AdamW optimizers became essential to modern LLM training. The post details how Adam, introduced in 2014 by Diederik Ki…
Researchers in the Philippines developed QHAdamW, a modified optimizer combining Quasi-Hyperbolic Momentum (QHAdam) and Adam with decoupled weight decay (AdamW), to train an artificial neural network …
Fine-tuning a 7B-parameter LLM requires 12 to 120 GB of VRAM depending on the method, according to a practical guide. Full fine-tuning in fp16 needs 80–120 GB, LoRA needs 24–32 GB, QLoRA needs 12–16 G…
In a technical explainer, the author demonstrates that L2 regularization and weight decay, equivalent in plain SGD, diverge under the Adam optimizer because Adam's adaptive step rescales the L2 penalt…
Researchers at NVIDIA have enhanced preconditioned gradient methods like SOAP and Muon to overcome computational cost and numerical stability challenges in large-scale LLM pretraining, demonstrating t…
Researchers introduced CvAdamW, a variant of the AdamW optimizer that monitors attention logit variance as specific heat to detect and accelerate grokking in neural networks, enabling generalization a…
A new study from researchers at arXiv finds that the Muon optimizer's speedup in reaching the grokking threshold on modular arithmetic comes from orthogonalization via the Newton-Schulz iteration, not…
A developer trained a 30M-parameter decoder-only transformer from scratch on the TinyStories dataset using Kaggle's free T4 GPUs, achieving a validation loss of 1.401 at a learning rate of 1e-3. The p…
A comprehensive guide to LLM fine-tuning covers full fine-tuning, LoRA, learning rate, and VRAM considerations, explaining the major stages of training an AI model from data preparation to production …
A technical blog post explains that AdamW with warmup and cosine decay is the optimizer almost everyone uses today to train large language models, tracing the evolution from vanilla SGD through moment…
A new study on arXiv (2607.13246v1) finds that Muon, an optimizer that reshapes gradient updates through approximate orthogonalization and has outperformed Adam and AdamW in large language model train…
A new study reveals that low-precision training can cause AI model parameters to freeze when gradient updates fall below half the unit in the last place (ULP) of a weight, effectively halting learning…
Researchers unveiled a latent world model using an equivariant encoder and predictor that achieves invariant predictions across orientations, with training loss symmetry ensuring consistent performanc…
Efficient Long-hOrizon (ELO) learning, a new learned optimizer, surpasses traditional models like AdamW and Muon across diverse tasks including language modeling and image classification, requiring le…
Researchers improved associative recall in recurrent neural networks by orthogonalizing the mLSTM memory matrix during reads, inspired by the Muon optimizer. In noisy associative recall tasks, the met…
A new character-level language model called EntropyBeam, using gradient-free count tables and a Dirichlet prior, achieved a validation loss of 1.596 nats on the Shakespeare character benchmark, outper…
TitanCore released Core-1, a distributed C++ AGI framework supporting up to 1 trillion parameters, featuring a 120-layer Mixture-of-Experts Transformer with full cognitive architecture including persi…
Ali, a 16-year-old independent researcher, developed PsiLogic, a chaos-aware optimizer built on Adam, and seeks an arXiv cs.LG endorser to submit his preprint. His benchmarks on an NVIDIA H100 GPU sho…