cd /news/machine-learning/classification-with-large-step-gradi… · home › topics › machine-learning › article
[ARTICLE · art-170319] src=arxiv.org ↗ pub= topic=machine-learning verified=true sentiment=· neutral

Classification with Large-Step Gradient Descent: Unifying Self-Stabilization and Convergence Rates

A new arXiv paper (2611.17459v1) extends a unified geometric theory of large-step gradient descent to exponential-tail losses, including logistic and cross-entropy losses in separable linear classification, where the loss infimum is never attained at a finite parameter. The authors identify a curvature-defined reference subspace that replaces the minimizer manifold used in prior overparameterized least-squares analysis, yielding a coupled dynamical system in which orthogonal oscillations are damped by parallel progress along directions of decreasing sharpness. Under regularity conditions verified for logistic and multiclass cross-entropy losses on symmetric data models, the paper derives explicit bounds on the iterates across both unstable and stable regimes, revealing a three-phase damping structure and a transient implicit bias in which large steps can leave persistent orthogonal residuals that affect finite-time alignment with the max-margin direction.

by read1 min views1 publishedNov 16, 2026

arXiv:2611.17459v1 Announce Type: new Abstract: Neural networks are often trained with gradient descent (GD) using step sizes that exceed classical stability thresholds, a regime where standard convergence theory no longer applies. Despite exhibiting non-monotonic loss and oscillatory dynamics, this regime frequently leads to faster convergence and better generalization. Existing theoretical perspectives offer complementary but incomplete insights: one identifies a self-stabilization mechanism that drives GD toward flat regions but does not yield convergence rates; another derives rigorous rates for specific losses but does not explain the underlying stabilization mechanism; and a third unifies these views for overparameterized least squares, but relies on the existence of a minimizer manifold, which is absent in classification settings. In this work, we extend this unified geometric perspective to exponential-tail losses, including logistic and cross-entropy losses in separable linear classification, where the loss infimum is not attained at any finite parameter. We identify a curvature-defined reference subspace that replaces the role of the minimizer manifold, yielding a coupled dynamical system in which orthogonal oscillations are damped by parallel progress along directions of decreasing sharpness. Under regularity conditions, which we verify for logistic and multiclass cross-entropy losses on symmetric data models, we derive explicit bounds for the iterates, not just the loss, covering both the unstable and stable regimes, and revealing a three-phase damping structure in the latter. Our analysis further uncovers a transient implicit bias induced by large step sizes: although GD converges asymptotically to the max-margin direction, large steps can leave persistent orthogonal residuals and, in certain geometries, amplify them, affecting finite-time alignment with the max-margin direction.

── more in #machine-learning 4 stories · sorted by recency
── more on @arxiv 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/classification-with-…] indexed:0 read:1min 2026-11-16 · —