Rethinking "RL Generalizes, SFT Memorizes": The Role of SFT Data A paper submitted November 11, 2026 to arXiv by Yunlong Hou and co-authors finds that supervised fine-tuning (SFT) can generalize as well as or better than reinforcement learning with verifiable rewards when SFT data is carefully mixed and scaled. On Qwen2.5 and Qwen3 models, matching the number of correct rollouts between SFT and RL greatly improved SFT generalization, and matching the total rollout budget let SFT generalize as well as RL; combining both factors let SFT outperform RL. The authors attribute the effect to larger datasets covering more tail solution methods, which they say provide generalizable reasoning signals, and support the findings with a theoretical analysis of shallow transformer training dynamics under both methods. Computer Science Machine Learning Submitted on 11 Nov 2026 v1 https://arxiv.org/abs/2611.14912v1 , last revised 17 Nov 2026 this version, v2 Title:Rethinking "RL Generalizes, SFT Memorizes": The Role of SFT Data View PDF https://arxiv.org/pdf/2611.14912 HTML experimental https://arxiv.org/html/2611.14912v2 Abstract:Large language models trained with Reinforcement Learning RL with verifiable rewards exhibit strong reasoning ability and broad generalization, whereas models trained with Supervised Fine-Tuning SFT are often viewed as more prone to memorization and limited transfer. This paper rethinks this distinction through the lens of SFT training data on widely used Qwen2.5 and Qwen3 series models. First, we study the role of data source and show that it is critical: a carefully mixed SFT dataset substantially outperforms data generated solely by a larger model. Second, we study the role of data scale and show that matching the number of correct rollouts between SFT and RL greatly improves SFT generalization, while matching the total rollout budget enables SFT to generalize as well as RL. Combining these two factors further enables SFT to generalize even better than RL. Third, by using LLM annotations to characterize the solution methods in training rollouts, we show that larger datasets cover more tail methods and that these tail methods provide generalizable reasoning signals. Finally, we support these empirical findings theoretically by analyzing the training dynamics of shallow transformers under both RL and SFT. Submission history From: Yunlong Hou view email https://arxiv.org/show-email/b1fa4386/2611.14912 Wed, 11 Nov 2026 02:10:24 UTC 1,870 KB \ v1\ https://arxiv.org/abs/2611.14912v1 v2 Tue, 17 Nov 2026 02:46:07 UTC 1,870 KB Additional Features References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender IArxiv Recommender What is IArxiv? https://iarxiv.org/about arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .