This could be the start of a trend in new pre-training recipes: No learning rate (LR) decay phase anymore. (Also seen in Kolibri.)
Instead, use a constant LR plus averaging the last N checkpoints (or a running weight average).
I guess that makes it easier in practice because you don’t have to commit to a total training length upfront.
A little meta analysis based on those who share training details.
- Classic cosine decay: Llama 3, OLMo 2, OLMo 3, GLM-4.5, GLM-5, Kimi K3, Phi-4.
- WSD (“Warmup-Stable-Decay”): DeepSeek V3, DeepSeek V4, DeepSeek V4.1-Flash, Kimi K2, Kimi Linear, Nemotron 3 Nano, Nemotron 3 Super, SmolLM3, MiniMax M2.
- No decay (constant LR plus checkpoint averaging): Kolibri-1 (see architecture in the LLM gallery ), DatologyAI’s models (see architecture in the LLM gallery ).
Source: lightly edited website version of my X post, quoting DatologyAI’s note on weight averaging.