cd /news/machine-learning/rethinking-rl-generalizes-sft-memori… · home › topics › machine-learning › article
[ARTICLE · art-172067] src=arxiv.org ↗ pub= topic=machine-learning verified=true sentiment=· neutral

Rethinking "RL Generalizes, SFT Memorizes": The Role of SFT Data

A paper submitted November 11, 2026 to arXiv by Yunlong Hou and co-authors finds that supervised fine-tuning (SFT) can generalize as well as or better than reinforcement learning with verifiable rewards when SFT data is carefully mixed and scaled. On Qwen2.5 and Qwen3 models, matching the number of correct rollouts between SFT and RL greatly improved SFT generalization, and matching the total rollout budget let SFT generalize as well as RL; combining both factors let SFT outperform RL. The authors attribute the effect to larger datasets covering more tail solution methods, which they say provide generalizable reasoning signals, and support the findings with a theoretical analysis of shallow transformer training dynamics under both methods.

recommendation

Before defaulting to RL for generalization, scale and mix SFT data to match RL's correct-rollout count and total rollout budget; the paper reports SFT then matches or beats RL on Qwen2.5 and Qwen3.

more in playbook: Fine-tuning and data
read2 min views3 publishedNov 18, 2026
Rethinking "RL Generalizes, SFT Memorizes": The Role of SFT Data
Image: source
  [Submitted on 11 Nov 2026 (

[v1](https://arxiv.org/abs/2611.14912v1)), last revised 17 Nov 2026 (this version, v2)]

[View PDF](https://arxiv.org/pdf/2611.14912)

[HTML (experimental)](https://arxiv.org/html/2611.14912v2)

Abstract:Large language models trained with Reinforcement Learning (RL) with verifiable rewards exhibit strong reasoning ability and broad generalization, whereas models trained with Supervised Fine-Tuning (SFT) are often viewed as more prone to memorization and limited transfer. This paper rethinks this distinction through the lens of SFT training data on widely used Qwen2.5 and Qwen3 series models. First, we study the role of data source and show that it is critical: a carefully mixed SFT dataset substantially outperforms data generated solely by a larger model. Second, we study the role of data scale and show that matching the number of correct rollouts between SFT and RL greatly improves SFT generalization, while matching the total rollout budget enables SFT to generalize as well as RL. Combining these two factors further enables SFT to generalize even better than RL. Third, by using LLM annotations to characterize the solution methods in training rollouts, we show that larger datasets cover more tail methods and that these tail methods provide generalizable reasoning signals. Finally, we support these empirical findings theoretically by analyzing the training dynamics of shallow transformers under both RL and SFT.

Submission history #

From: Yunlong Hou [
[view email](https://arxiv.org/show-email/b1fa4386/2611.14912)]

**Wed, 11 Nov 2026 02:10:24 UTC (1,870 KB)**

[\[v1\]](https://arxiv.org/abs/2611.14912v1)
**[v2]** Tue, 17 Nov 2026 02:46:07 UTC (1,870 KB)

Additional Features

References & Citations

...

Bibliographic Explorer

(What is the Explorer?) Connected Papers

(What is Connected Papers?) Litmaps

(What is Litmaps?) scite Smart Citations

(What are Smart Citations?) alphaXiv

(What is alphaXiv?) CatalyzeX Code Finder for Papers

(What is CatalyzeX?) DagsHub

(What is DagsHub?) Gotit.pub

(What is GotitPub?) Hugging Face

(What is Huggingface?) ScienceCast

(What is ScienceCast?) Influence Flower

(What are Influence Flowers?) CORE Recommender

(What is CORE?) IArxiv Recommender

(What is IArxiv?) arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

── more in #machine-learning 4 stories · sorted by recency
── more on @yunlong hou 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/rethinking-rl-genera…] indexed:0 read:2min 2026-11-18 · —