{"slug": "decentralized-sgd-under-heavy-tailed-noise-optimal-convergence-rates-and-the-of", "title": "Decentralized SGD under Heavy-Tailed Noise: Optimal Convergence Rates and the Role of Gradient Clipping", "summary": "A paper submitted to arXiv on 7 October 2026 by Aleksandar Armacki shows that clipped decentralized SGD (DSGD) achieves order-optimal convergence rates under heavy-tailed noise for smooth non-convex costs with bounded p-th moment noise, p in (1,2], both with high probability and in expectation. The analysis establishes a linear speed-up in the number of agents and exploits the structure of gradient clipping to relegate network effects to higher-order terms, distinguishing clipping from normalization, which can fail to converge in decentralized settings because it discards gradient magnitude information. Numerical experiments validate the theory.", "body_md": "# Mathematics > Optimization and Control\n\n  [Submitted on 7 Oct 2026]\n\n# Title:Decentralized SGD under Heavy-Tailed Noise: Optimal Convergence Rates and the Role of Gradient Clipping\n\n[View PDF](http://arxiv.org/pdf/2610.10527v1)\n\n[HTML (experimental)](https://arxiv.org/html/2610.10527v1)\n\nAbstract:Heavy-tailed noise has been widely observed in modern machine learning, motivating the use of methods like gradient clipping and normalization. While these methods are well understood in centralized settings, much less is known in decentralized ones, where applying a nonlinearity to local gradients affects both optimization and consensus. Recent works on decentralized non-convex optimization have studied both clipping and normalization under heavy-tailed noise, with clipping yielding suboptimal rates and normalization needing local momentum or mini-batches to converge. This raises the question: can a baseline decentralized method using a nonlinearity achieve optimal convergence rates under heavy-tailed noise? We answer affirmatively with clipped decentralized SGD ($\\mathtt{DSGD}$). For smooth non-convex costs under bounded $p$-th moment noise, $p \\in (1,2]$, we show that clipped $\\mathtt{DSGD}$ achieves order-optimal rates both with high probability and in expectation. Moreover, we establish a linear speed-up in the number of agents, which, to our knowledge, has not been shown for decentralized methods with clipping. The key technical ingredient is a sharp analysis of the consensus gap that exploits the structure of clipping, relegating network effects to higher-order terms. Our results highlight an important distinction between clipping and normalization in decentralized settings: while normalized $\\mathtt{DSGD}$ can fail to converge, clipping retains magnitude information, enabling $\\mathtt{DSGD}$ to be convergent and order-optimal. Numerical experiments validate our theory.\n    \n\n## Submission history\n\nFrom: Aleksandar Armacki [\n[view email](http://arxiv.org/show-email/df2e21c4/2610.10527)]\n\n**[v1]** Wed, 7 Oct 2026 17:57:58 UTC (1,521 KB)\n\n### Current browse context:\n\nmath.OC\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/decentralized-sgd-under-heavy-tailed-noise-optimal-convergence-rates-and-the-of", "canonical_source": "http://arxiv.org/abs/2610.10527v1", "published_at": "2026-10-08 14:26:05+00:00", "updated_at": "2026-10-08 14:49:33.837330+00:00", "lang": "en", "topics": ["machine-learning", "ai-research"], "entities": ["arXiv", "Aleksandar Armacki", "clipped decentralized SGD", "DSGD"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/decentralized-sgd-under-heavy-tailed-noise-optimal-convergence-rates-and-the-of", "markdown": "https://wpnews.pro/news/decentralized-sgd-under-heavy-tailed-noise-optimal-convergence-rates-and-the-of.md", "text": "https://wpnews.pro/news/decentralized-sgd-under-heavy-tailed-noise-optimal-convergence-rates-and-the-of.txt", "jsonld": "https://wpnews.pro/news/decentralized-sgd-under-heavy-tailed-noise-optimal-convergence-rates-and-the-of.jsonld"}}