Towards Understanding Momentum Acceleration in River-Valley Loss Landscape
A new arXiv paper (2609.30957v1) establishes a theoretical analysis showing that momentum accelerates optimization in the "river-valley" loss landscape of large language model pretraining by stabilizi…