{"slug": "curvature-conditioned-multiscale-momentum-with-sphere-constraints-for-llm", "title": "Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining", "summary": "Researchers propose a curvature-conditioned multiscale momentum method with sphere constraints that accelerates LLM pretraining, improving Muon across dense and MoE architectures from 0.12B to 2.3B parameters. The method targets flat directions to speed up final loss reduction, addressing ill-conditioned loss landscapes.", "body_md": "arXiv:2608.28442v1 Announce Type: new\nAbstract: Pretraining accounts for a large fraction of the total computational cost in LLM training. However, noise-dominant gradients and the highly ill-conditioned loss landscape bring severe challenges. Although modern adaptive optimizers such as AdamW and Muon have achieved great success in large-scale pretraining, their reliance on gradient normalization offers limited mitigation of the ill-conditioned curvature. The progress along flat directions (eigen-directions of small eigenvalues), which dominates the final loss reduction, remains relatively slow. To enhance training dynamics along flat directions, we propose a curvature-conditioned multiscale momentum method with sphere constraints, delivering steady acceleration in LLM pretraining. This multiscale momentum, applied only along flat directions, pairs a slow-decay component for noise reduction with a fast-decay component for rapid curvature adaptation, harnessing their complementary strengths. Crucially, we employ a sphere constraint technique to prevent parameter inflation and excessively rapid effective learning rate decay that would otherwise arise from a naive combination. Extensive experiments show that the proposed method significantly accelerates Muon across diverse architectures (dense, MoE) and model sizes (0.12B--2.3B parameters). Theoretically, we verify the acceleration effect and provide insight into the design principles underlying the flat-direction multiscale momentum.", "url": "https://wpnews.pro/news/curvature-conditioned-multiscale-momentum-with-sphere-constraints-for-llm", "canonical_source": "https://www.machinebrief.com/news/curvature-conditioned-multiscale-momentum-with-sphere-constr-nqrm", "published_at": "2026-08-31 04:00:00+00:00", "updated_at": "2026-08-31 05:52:05.940462+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research"], "entities": ["Muon", "AdamW", "arXiv"], "alternates": {"html": "https://wpnews.pro/news/curvature-conditioned-multiscale-momentum-with-sphere-constraints-for-llm", "markdown": "https://wpnews.pro/news/curvature-conditioned-multiscale-momentum-with-sphere-constraints-for-llm.md", "text": "https://wpnews.pro/news/curvature-conditioned-multiscale-momentum-with-sphere-constraints-for-llm.txt", "jsonld": "https://wpnews.pro/news/curvature-conditioned-multiscale-momentum-with-sphere-constraints-for-llm.jsonld"}}