cd /news/machine-learning/pre-training-without-learning-rate-d… · home › topics › machine-learning › article
[ARTICLE · art-171349] src=sebastianraschka.com ↗ pub= topic=machine-learning verified=true sentiment=· neutral

Pre-Training Without Learning Rate Decay

A growing number of pre-training recipes are dropping the learning rate decay phase in favor of a constant learning rate plus averaging the last N checkpoints, according to a blog post by Sebastian Raschka citing DatologyAI's note on weight averaging. Raschka's survey of disclosed training details places Llama 3, OLMo 2, OLMo 3, GLM-4.5, GLM-5, Kimi K3 and Phi-4 in the classic cosine-decay camp; DeepSeek V3, DeepSeek V4, DeepSeek V4.1-Flash, Kimi K2, Kimi Linear, Nemotron 3 Nano, Nemotron 3 Super, SmolLM3 and MiniMax M2 in the warmup-stable-decay camp; and Kolibri-1 and DatologyAI's models in the no-decay camp. The approach removes the need to commit to a total training length upfront.

key takeaway

Constant learning rate with averaging of the last N checkpoints (or a running weight average) replaces cosine or warmup-stable-decay schedules in Kolibri-1 and DatologyAI's models, letting teams avoid committing to a total training length upfront.

read1 min views2 publishedNov 17, 2026
Pre-Training Without Learning Rate Decay
Image: Sebastianraschka (auto-discovered)

This could be the start of a trend in new pre-training recipes: No learning rate (LR) decay phase anymore. (Also seen in Kolibri.)

Instead, use a constant LR plus averaging the last N checkpoints (or a running weight average).

I guess that makes it easier in practice because you don’t have to commit to a total training length upfront.

A little meta analysis based on those who share training details.

  • Classic cosine decay: Llama 3, OLMo 2, OLMo 3, GLM-4.5, GLM-5, Kimi K3, Phi-4.
  • WSD (“Warmup-Stable-Decay”): DeepSeek V3, DeepSeek V4, DeepSeek V4.1-Flash, Kimi K2, Kimi Linear, Nemotron 3 Nano, Nemotron 3 Super, SmolLM3, MiniMax M2.
  • No decay (constant LR plus checkpoint averaging): Kolibri-1 (see architecture in the LLM gallery ), DatologyAI’s models (see architecture in the LLM gallery ).

Source: lightly edited website version of my X post, quoting DatologyAI’s note on weight averaging.

From Full to Hybrid Attention in LLMs How open-weight LLMs are shifting from full attention to hybrid and sparse designs, and what that means for long-context compute and memory.

Open October LLM Architecture Roundup A roundup of ten open-weight LLM architecture releases from October 2026, covering attention mechanisms, hybrid layers, and MoE designs.

Focusing on Post-Training Why I'd invest in post-training existing open-weight LLMs, with Fireworks' Ember-1 as an example of more token-efficient reasoning.

── more in #machine-learning 4 stories · sorted by recency
── more on @sebastian raschka 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/pre-training-without…] indexed:0 read:1min 2026-11-17 · —