Pre-Training Without Learning Rate Decay A growing number of pre-training recipes are dropping the learning rate decay phase in favor of a constant learning rate plus averaging the last N checkpoints, according to a blog post by Sebastian Raschka citing DatologyAI's note on weight averaging. Raschka's survey of disclosed training details places Llama 3, OLMo 2, OLMo 3, GLM-4.5, GLM-5, Kimi K3 and Phi-4 in the classic cosine-decay camp; DeepSeek V3, DeepSeek V4, DeepSeek V4.1-Flash, Kimi K2, Kimi Linear, Nemotron 3 Nano, Nemotron 3 Super, SmolLM3 and MiniMax M2 in the warmup-stable-decay camp; and Kolibri-1 and DatologyAI's models in the no-decay camp. The approach removes the need to commit to a total training length upfront. Pre-Training Without Learning Rate Decay This could be the start of a trend in new pre-training recipes: No learning rate LR decay phase anymore. Also seen in Kolibri. Instead, use a constant LR plus averaging the last N checkpoints or a running weight average . I guess that makes it easier in practice because you don’t have to commit to a total training length upfront. A little meta analysis based on those who share training details. - Classic cosine decay: Llama 3, OLMo 2, OLMo 3, GLM-4.5, GLM-5, Kimi K3, Phi-4. - WSD “Warmup-Stable-Decay” : DeepSeek V3, DeepSeek V4, DeepSeek V4.1-Flash, Kimi K2, Kimi Linear, Nemotron 3 Nano, Nemotron 3 Super, SmolLM3, MiniMax M2. - No decay constant LR plus checkpoint averaging : Kolibri-1 see architecture in the LLM gallery https://sebastianraschka.com/llm-architecture-gallery/ card-kolibri-1-78b , DatologyAI’s models see architecture in the LLM gallery https://sebastianraschka.com/llm-architecture-gallery/ card-datology-xl-30b-a3b . Source: lightly edited website version of my X post https://x.com/rasbt/status/2122721652433715481 , quoting DatologyAI’s note on weight averaging https://x.com/datologyai/status/2122366636472869015 . Read Next From Full to Hybrid Attention in LLMs How open-weight LLMs are shifting from full attention to hybrid and sparse designs, and what that means for long-context compute and memory. https://sebastianraschka.com/blog/2026/full-to-hybrid-attention.html Open October LLM Architecture Roundup A roundup of ten open-weight LLM architecture releases from October 2026, covering attention mechanisms, hybrid layers, and MoE designs. https://sebastianraschka.com/blog/2026/open-october-llm-architecture-releases.html Focusing on Post-Training Why I'd invest in post-training existing open-weight LLMs, with Fireworks' Ember-1 as an example of more token-efficient reasoning. https://sebastianraschka.com/blog/2026/focusing-on-llm-post-training.html