{"slug": "rethinking-rl-generalizes-sft-memorizes-the-role-of-sft-data", "title": "Rethinking \"RL Generalizes, SFT Memorizes\": The Role of SFT Data", "summary": "A paper submitted November 11, 2026 to arXiv by Yunlong Hou and co-authors finds that supervised fine-tuning (SFT) can generalize as well as or better than reinforcement learning with verifiable rewards when SFT data is carefully mixed and scaled. On Qwen2.5 and Qwen3 models, matching the number of correct rollouts between SFT and RL greatly improved SFT generalization, and matching the total rollout budget let SFT generalize as well as RL; combining both factors let SFT outperform RL. The authors attribute the effect to larger datasets covering more tail solution methods, which they say provide generalizable reasoning signals, and support the findings with a theoretical analysis of shallow transformer training dynamics under both methods.", "body_md": "# Computer Science > Machine Learning\n\n  [Submitted on 11 Nov 2026 (\n\n[v1](https://arxiv.org/abs/2611.14912v1)), last revised 17 Nov 2026 (this version, v2)]\n# Title:Rethinking \"RL Generalizes, SFT Memorizes\": The Role of SFT Data\n\n[View PDF](https://arxiv.org/pdf/2611.14912)\n\n[HTML (experimental)](https://arxiv.org/html/2611.14912v2)\n\nAbstract:Large language models trained with Reinforcement Learning (RL) with verifiable rewards exhibit strong reasoning ability and broad generalization, whereas models trained with Supervised Fine-Tuning (SFT) are often viewed as more prone to memorization and limited transfer. This paper rethinks this distinction through the lens of SFT training data on widely used Qwen2.5 and Qwen3 series models. First, we study the role of data source and show that it is critical: a carefully mixed SFT dataset substantially outperforms data generated solely by a larger model. Second, we study the role of data scale and show that matching the number of correct rollouts between SFT and RL greatly improves SFT generalization, while matching the total rollout budget enables SFT to generalize as well as RL. Combining these two factors further enables SFT to generalize even better than RL. Third, by using LLM annotations to characterize the solution methods in training rollouts, we show that larger datasets cover more tail methods and that these tail methods provide generalizable reasoning signals. Finally, we support these empirical findings theoretically by analyzing the training dynamics of shallow transformers under both RL and SFT.\n    \n\n## Submission history\n\nFrom: Yunlong Hou [\n[view email](https://arxiv.org/show-email/b1fa4386/2611.14912)]\n\n**Wed, 11 Nov 2026 02:10:24 UTC (1,870 KB)**\n\n[\\[v1\\]](https://arxiv.org/abs/2611.14912v1)\n**[v2]** Tue, 17 Nov 2026 02:46:07 UTC (1,870 KB)\n\n### Additional Features\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\nIArxiv Recommender\n\n*(*[What is IArxiv?](https://iarxiv.org/about))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/rethinking-rl-generalizes-sft-memorizes-the-role-of-sft-data", "canonical_source": "https://arxiv.org/abs/2611.14912", "published_at": "2026-11-18 18:59:27+00:00", "updated_at": "2026-11-18 19:22:21.885564+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "natural-language-processing"], "entities": ["Yunlong Hou", "arXiv", "Qwen2.5", "Qwen3"], "also_reported_by": [], "takeaway": "Matching SFT's correct-rollout count to RL's and then matching the total rollout budget let SFT generalize as well as RL on Qwen2.5 and Qwen3, and combining both factors made SFT generalize better.", "recommendation": {"advice": "Before defaulting to RL for generalization, scale and mix SFT data to match RL's correct-rollout count and total rollout budget; the paper reports SFT then matches or beats RL on Qwen2.5 and Qwen3.", "playbook": "training-data", "playbook_url": "https://wpnews.pro/playbooks/training-data"}, "alternates": {"html": "https://wpnews.pro/news/rethinking-rl-generalizes-sft-memorizes-the-role-of-sft-data", "markdown": "https://wpnews.pro/news/rethinking-rl-generalizes-sft-memorizes-the-role-of-sft-data.md", "text": "https://wpnews.pro/news/rethinking-rl-generalizes-sft-memorizes-the-role-of-sft-data.txt", "jsonld": "https://wpnews.pro/news/rethinking-rl-generalizes-sft-memorizes-the-role-of-sft-data.jsonld"}}