{"slug": "the-convergence-behavior-of-adam-under-heavy-tailed-noise", "title": "The Convergence Behavior of Adam under Heavy-Tailed Noise", "summary": "Researchers at arXiv (paper 2607.27383v1) established the first convergence guarantees for the plain vector-form Adam optimizer under heavy-tailed stochastic noise, showing it converges to (ρ,ε)-stationary points but with suboptimal iteration complexity that persists even in bounded-variance cases (p=2). When the domain radius is known, the convergence rate improves to match optimal complexity, providing new theoretical insight into Adam's robustness and limitations in heavy-tailed regimes.", "body_md": "arXiv:2607.27383v1 Announce Type: new\nAbstract: We establish the first convergence guarantees for the plain vector-form \\emph{Adam} optimizer under heavy-tailed stochastic noise. While several Adam variants are known to achieve optimal iteration complexity in bounded-variance nonconvex optimization, little is understood about their behavior when stochastic gradients admit only a bounded $p$-th central moment for some $p \\in (1,2]$, a setting increasingly observed in modern deep learning. To address this gap, we generalize the recent online-to-nonconvex conversion framework to accommodate heavy-tailed martingale-difference noise. Building on this generalized framework, we develop a discounted regret analysis for Adam, without restrictive parameter coupling. Our results show that Adam converges to $(\\rho,\\epsilon)$-stationary points under heavy-tailed noise. However, it exhibits a suboptimal iteration complexity and $p$-dependent convergence, a suboptimality that persists even in the bounded-variance case ($p=2$). When the domain radius is known and used to control the online-learner output, a standard setup in related literature, the convergence rate improves to match the optimal complexity. These findings provide new theoretical insight into the robustness and limitations of Adam in heavy-tailed regimes.", "url": "https://wpnews.pro/news/the-convergence-behavior-of-adam-under-heavy-tailed-noise", "canonical_source": "https://arxiv.org/abs/2607.27383", "published_at": "2026-07-31 04:00:00+00:00", "updated_at": "2026-07-31 04:35:10.559514+00:00", "lang": "en", "topics": ["machine-learning"], "entities": ["arXiv"], "alternates": {"html": "https://wpnews.pro/news/the-convergence-behavior-of-adam-under-heavy-tailed-noise", "markdown": "https://wpnews.pro/news/the-convergence-behavior-of-adam-under-heavy-tailed-noise.md", "text": "https://wpnews.pro/news/the-convergence-behavior-of-adam-under-heavy-tailed-noise.txt", "jsonld": "https://wpnews.pro/news/the-convergence-behavior-of-adam-under-heavy-tailed-noise.jsonld"}}