cd /news/machine-learning/model-annealing-data-efficient-learn… · home › topics › machine-learning › article
[ARTICLE · art-170288] src=arxiv.org ↗ pub= topic=machine-learning verified=true sentiment=↑ positive

Model Annealing: Data-Efficient Learning through Review and Consolidation

A new arXiv paper (2611.17524v1) reports that model annealing, a training method that spends leftover budget relearning from the network's own tempered predictions, raised top-1 accuracy over cross-entropy training in 28 of 33 configurations across CIFAR-100, Tiny-ImageNet and Imagewoof, by 1.3 points on average, for AdamW-trained ConvNeXt, ViT and MLP-Mixer models at 5% to 100% of labels. The method matches snapshot distillation on accuracy and beats it on likelihood, calibration and robustness to corruptions and small adversarial perturbations, while for an SGD-trained ResNet-18 it lowers accuracy but still improves robustness to small adversarial perturbations. The authors attribute the gains to relearning and the losses to reheats, and report that a memory of past cooled states plus adversarial dreams together remove the overconfidence drift seen with snapshot distillation at low label counts.

by read1 min views3 publishedNov 16, 2026

arXiv:2611.17524v1 Announce Type: new Abstract: A small labelled dataset is memorized long before the training budget runs out, and the one-hot labels that remain to be fitted then mostly sharpen confidence. Model annealing spends the rest of the budget on review and consolidation, relearning from the network's own knowledge. Training is divided into episodes whose learning rate is raised again and cooled, every cooled state joins a memory of past selves, and each new episode learns from the labels, from the tempered predictions of this memory, and from adversarial dreams sought where student and memory disagree most. The memory averages predictions rather than weights, and the deployed network is the last student, so inference costs nothing extra. Across CIFAR-100, Tiny-ImageNet and Imagewoof, four architecture families and 5% to 100% of the labels, model annealing raises the top-1 accuracy of the AdamW-trained ConvNeXt, ViT and MLP-Mixer over cross-entropy training in 28 of 33 configurations, by 1.3 points on average, and is on par in accuracy with snapshot distillation, the closest competitor. On these families it is better than both in likelihood, calibration and robustness to corruptions and small adversarial perturbations. For an SGD-trained ResNet-18 it lowers accuracy but still improves robustness to small adversarial perturbations. Separating the reheated schedule from relearning shows that relearning helps every family and that the losses come from reheats, which undo most of this network's fit to its training set. How early the accuracy of a cross-entropy run saturates largely separates the AdamW families from ResNet-18 but does not predict the gain within them. With little data, relearning from the latest snapshot without dreams, as snapshot distillation does, compounds overconfidence in the AdamW families from episode to episode. The memory and the dreams each curb most of this drift, and together they remove it.

── more in #machine-learning 4 stories · sorted by recency
── more on @arxiv 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/model-annealing-data…] indexed:0 read:1min 2026-11-16 · —