{"slug": "rl-for-llm-reasoning-is-sparse-policy-selection-not-capability-learning", "title": "RL for LLM Reasoning Is Sparse Policy Selection, Not Capability Learning", "summary": "A new arXiv preprint (2605.06241v2) finds that reinforcement learning (RL) for large language model reasoning acts as sparse policy selection rather than capability learning, with only 1–3% of token positions affected and promoted tokens always within the base model's top-5 alternatives. The authors introduce ReasonMaxxer, an RL-free method applying contrastive loss only at entropy-gated decision points, which matches or exceeds full RL performance across three model families, six scales, and six math benchmarks while requiring only tens of problems and minutes of single-GPU training—a roughly three orders of magnitude reduction in training cost.", "body_md": "# Computer Science > Computation and Language\n\n[Submitted on 7 May 2026 (\n\n[v1](https://arxiv.org/abs/2605.06241v1)), last revised 8 May 2026 (this version, v2)]# Title:Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning\n\n[View PDF](/pdf/2605.06241)\n\n[HTML (experimental)](https://arxiv.org/html/2605.06241v2)\n\nAbstract:Reinforcement learning has become the standard for improving reasoning in large language models, yet evidence increasingly suggests that RL does not teach new strategies; it redistributes probability mass over solutions the base model already contains. In this work, we ask: if RL merely steers the model toward paths it already knows, is the RL optimization loop itself necessary? Through token-level analysis across multiple model families and RL algorithms, we find that RL's beneficial footprint is a sparse, predictable correction concentrated at high-entropy decision points where the model is uncertain which branch to take. Only 1--3\\% of token positions are affected, the promoted token always lies within the base model's top-5 alternatives, and targeted corrections at those few positions causally recover a large fraction of RL's accuracy gain, while random corrections fail. The base model's own entropy identifies these positions without any RL-trained model, and the entire correction is low-dimensional, representable in a tiny fraction of model parameters. These findings reframe reasoning improvement as sparse policy selection, not capability acquisition. We translate this insight into ReasonMaxxer, a minimal RL-free method that applies contrastive loss only at entropy-gated decision points, using a few hundred base-model rollouts and no online generation. Across three model families, six scales, and six math reasoning benchmarks, ReasonMaxxer matches or exceeds full RL performance while requiring only tens of problems and minutes of single-GPU training, a reduction in training cost of roughly three orders of magnitude.\n\n## Submission history\n\nFrom: Ömer Faruk Akgül [[view email](/show-email/2ef80c8d/2605.06241)]\n\n**Thu, 7 May 2026 13:25:05 UTC (438 KB)**\n\n[[v1]](/abs/2605.06241v1)**[v2]** Fri, 8 May 2026 19:48:19 UTC (438 KB)\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer\n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers\n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps\n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations\n\n*(*[What are Smart Citations?](https://www.scite.ai/))# Code, Data and Media Associated with this Article\n\nalphaXiv\n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers\n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub\n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub\n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face\n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast\n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower\n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender\n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [ Learn more about arXivLabs](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/rl-for-llm-reasoning-is-sparse-policy-selection-not-capability-learning", "canonical_source": "https://arxiv.org/abs/2605.06241", "published_at": "2026-08-16 13:32:41+00:00", "updated_at": "2026-08-16 13:40:54.093339+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["arXiv", "ReasonMaxxer", "Ömer Faruk Akgül"], "alternates": {"html": "https://wpnews.pro/news/rl-for-llm-reasoning-is-sparse-policy-selection-not-capability-learning", "markdown": "https://wpnews.pro/news/rl-for-llm-reasoning-is-sparse-policy-selection-not-capability-learning.md", "text": "https://wpnews.pro/news/rl-for-llm-reasoning-is-sparse-policy-selection-not-capability-learning.txt", "jsonld": "https://wpnews.pro/news/rl-for-llm-reasoning-is-sparse-policy-selection-not-capability-learning.jsonld"}}