{"slug": "optimization-as-a-dynamical-system-generative-schedules-from-latent-odes", "title": "Optimization as a Dynamical System: Generative Schedules from Latent ODEs", "summary": "Researchers present a meta-learning method that models gradient descent training as a dynamical system using latent ordinary differential equations (ODEs) to predict optimal learning rate schedules. The method outperforms all baselines on image classification with CNN and ResNet models and next-token prediction with a transformer, yielding models in flatter loss regions for better generalization. It is computationally efficient, optimizer-agnostic, and integrates with ML experiment-tracking platforms.", "body_md": "arXiv:2509.23052v2 Announce Type: replace\nAbstract: We present a new meta-learning method to determine the optimal learning rate schedule for gradient descent. It leverages training runs from a hyperparameter search to learn a latent representation of the training process, which is modeled as a dynamical system. Given current training metrics, it predicts the future learning rate schedule with the best long-term validation performance. Our scheduler generalizes beyond previously observed training dynamics and creates specialized schedules that deviate noticeably from even the best-performing parametric functions. It outperforms all baselines we compare to on results for image classification with CNN and ResNet models as well as for next-token prediction with a transformer model. The trained models are located in flatter regions of the loss landscape and thus provide better generalization than those trained with other schedules. Our method is computationally efficient, optimizer-agnostic, and can easily be layered on top of ML experiment-tracking platforms to streamline training of neural networks from scratch.", "url": "https://wpnews.pro/news/optimization-as-a-dynamical-system-generative-schedules-from-latent-odes", "canonical_source": "https://www.machinebrief.com/news/optimization-as-a-dynamical-system-generative-schedules-from-f83s", "published_at": "2026-08-10 04:00:00+00:00", "updated_at": "2026-08-10 05:12:19.727894+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence", "neural-networks"], "entities": ["arXiv", "CNN", "ResNet", "transformer"], "alternates": {"html": "https://wpnews.pro/news/optimization-as-a-dynamical-system-generative-schedules-from-latent-odes", "markdown": "https://wpnews.pro/news/optimization-as-a-dynamical-system-generative-schedules-from-latent-odes.md", "text": "https://wpnews.pro/news/optimization-as-a-dynamical-system-generative-schedules-from-latent-odes.txt", "jsonld": "https://wpnews.pro/news/optimization-as-a-dynamical-system-generative-schedules-from-latent-odes.jsonld"}}