Researchers derived a three-force decomposition of AdamW training dynamics explaining Weibull weight-scale parameter evolution in transformers, finding alignment force dominates growth. A spline displ…
A 124M-parameter GPT-2 model trained from scratch on OpenWebText data using a custom deep learning library achieved a validation loss of 2.764 nats and a perplexity of 15.87 after 56,000 steps (27.5B …
Recursive's automated AI research system achieved state-of-the-art results on three benchmarks: fixed-budget language model training, small-model training speed, and GPU kernel optimization. The syste…
Researchers propose Gefen, a memory-efficient optimizer that reduces AdamW's memory footprint by ~8x while maintaining performance, enabling larger microbatches and improved throughput in deep learnin…
DeepSpeed has integrated the Muon Optimizer, a memory-efficient optimizer that uses a single momentum buffer and Newton-Schulz orthogonalization to improve training convergence, particularly for 2D we…
A new study proves the Spectral Alignment Decomposition, which explains why the curvature exponent $\alpha$ — governing how Hessian eigenvalues scale with gradient singular values — varies across neur…