{"slug": "pre-training-without-learning-rate-decay", "title": "Pre-Training Without Learning Rate Decay", "summary": "A growing number of pre-training recipes are dropping the learning rate decay phase in favor of a constant learning rate plus averaging the last N checkpoints, according to a blog post by Sebastian Raschka citing DatologyAI's note on weight averaging. Raschka's survey of disclosed training details places Llama 3, OLMo 2, OLMo 3, GLM-4.5, GLM-5, Kimi K3 and Phi-4 in the classic cosine-decay camp; DeepSeek V3, DeepSeek V4, DeepSeek V4.1-Flash, Kimi K2, Kimi Linear, Nemotron 3 Nano, Nemotron 3 Super, SmolLM3 and MiniMax M2 in the warmup-stable-decay camp; and Kolibri-1 and DatologyAI's models in the no-decay camp. The approach removes the need to commit to a total training length upfront.", "body_md": "# Pre-Training Without Learning Rate Decay\n\nThis could be the start of a trend in new pre-training recipes: No learning rate (LR) decay phase anymore. (Also seen in Kolibri.)\n\nInstead, use a constant LR plus averaging the last N checkpoints (or a running weight average).\n\nI guess that makes it easier in practice because you don’t have to commit to a total training length upfront.\n\nA little meta analysis based on those who share training details.\n\n- Classic cosine decay: Llama 3, OLMo 2, OLMo 3, GLM-4.5, GLM-5, Kimi K3, Phi-4.\n- WSD (“Warmup-Stable-Decay”): DeepSeek V3, DeepSeek V4, DeepSeek V4.1-Flash, Kimi K2, Kimi Linear, Nemotron 3 Nano, Nemotron 3 Super, SmolLM3, MiniMax M2.\n- No decay (constant LR plus checkpoint averaging): Kolibri-1 ([see architecture in the LLM gallery](https://sebastianraschka.com/llm-architecture-gallery/#card-kolibri-1-78b) ), DatologyAI’s models ([see architecture in the LLM gallery](https://sebastianraschka.com/llm-architecture-gallery/#card-datology-xl-30b-a3b) ).\n\nSource: lightly edited website version of my [X post](https://x.com/rasbt/status/2122721652433715481), quoting [DatologyAI’s note on weight averaging](https://x.com/datologyai/status/2122366636472869015).\n\n## Read Next\n\n[From Full to Hybrid Attention in LLMs How open-weight LLMs are shifting from full attention to hybrid and sparse designs, and what that means for long-context compute and memory.](https://sebastianraschka.com/blog/2026/full-to-hybrid-attention.html)\n\n[Open October LLM Architecture Roundup A roundup of ten open-weight LLM architecture releases from October 2026, covering attention mechanisms, hybrid layers, and MoE designs.](https://sebastianraschka.com/blog/2026/open-october-llm-architecture-releases.html)\n\n[Focusing on Post-Training Why I'd invest in post-training existing open-weight LLMs, with Fireworks' Ember-1 as an example of more token-efficient reasoning.](https://sebastianraschka.com/blog/2026/focusing-on-llm-post-training.html)", "url": "https://wpnews.pro/news/pre-training-without-learning-rate-decay", "canonical_source": "https://sebastianraschka.com/blog/2026/llm-pretraining-without-learning-rate-decay.html", "published_at": "2026-11-17 16:00:54+00:00", "updated_at": "2026-11-17 21:20:55.642538+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research"], "entities": ["Sebastian Raschka", "DatologyAI", "Kolibri-1", "Llama 3", "DeepSeek V3", "Kimi K2", "SmolLM3", "Nemotron 3"], "also_reported_by": [], "takeaway": "Constant learning rate with averaging of the last N checkpoints (or a running weight average) replaces cosine or warmup-stable-decay schedules in Kolibri-1 and DatologyAI's models, letting teams avoid committing to a total training length upfront.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/pre-training-without-learning-rate-decay", "markdown": "https://wpnews.pro/news/pre-training-without-learning-rate-decay.md", "text": "https://wpnews.pro/news/pre-training-without-learning-rate-decay.txt", "jsonld": "https://wpnews.pro/news/pre-training-without-learning-rate-decay.jsonld"}}