A developer published a comprehensive guide to training large language models, covering stages from data preparation and tokenizer selection to pretraining, LoRA, RLHF, evaluation, and production moni…
Optimizers like Momentum, Adam, and learning rate schedules solve problems with vanilla SGD, such as zigzagging in steep valleys and slow progress in flat directions. Momentum adds inertia to smooth o…
Researchers at arXiv found that Muon, an optimizer that outperforms Adam and AdamW in large language model training, does not consistently beat AdamW in low-rank matrix factorization, a simpler contro…
A new study reveals that low-precision training can cause AI model parameters to freeze when gradient updates fall below half the unit in the last place (ULP) of a weight, effectively halting learning…
Researchers unveiled a latent world model using an equivariant encoder and predictor that achieves invariant predictions across orientations, with training loss symmetry ensuring consistent performanc…
Efficient Long-hOrizon (ELO) learning, a new learned optimizer, surpasses traditional models like AdamW and Muon across diverse tasks including language modeling and image classification, requiring le…
Researchers improved associative recall in recurrent neural networks by orthogonalizing the mLSTM memory matrix during reads, inspired by the Muon optimizer. In noisy associative recall tasks, the met…
A new character-level language model called EntropyBeam, using gradient-free count tables and a Dirichlet prior, achieved a validation loss of 1.596 nats on the Shakespeare character benchmark, outper…
TitanCore released Core-1, a distributed C++ AGI framework supporting up to 1 trillion parameters, featuring a 120-layer Mixture-of-Experts Transformer with full cognitive architecture including persi…
Ali, a 16-year-old independent researcher, developed PsiLogic, a chaos-aware optimizer built on Adam, and seeks an arXiv cs.LG endorser to submit his preprint. His benchmarks on an NVIDIA H100 GPU sho…
Researchers derived a three-force decomposition of AdamW training dynamics explaining Weibull weight-scale parameter evolution in transformers, finding alignment force dominates growth. A spline displ…
A 124M-parameter GPT-2 model trained from scratch on OpenWebText data using a custom deep learning library achieved a validation loss of 2.764 nats and a perplexity of 15.87 after 56,000 steps (27.5B …
Recursive's automated AI research system achieved state-of-the-art results on three benchmarks: fixed-budget language model training, small-model training speed, and GPU kernel optimization. The syste…
Researchers propose Gefen, a memory-efficient optimizer that reduces AdamW's memory footprint by ~8x while maintaining performance, enabling larger microbatches and improved throughput in deep learnin…
DeepSpeed has integrated the Muon Optimizer, a memory-efficient optimizer that uses a single momentum buffer and Newton-Schulz orthogonalization to improve training convergence, particularly for 2D we…
A new study proves the Spectral Alignment Decomposition, which explains why the curvature exponent $\alpha$ — governing how Hessian eigenvalues scale with gradient singular values — varies across neur…