Extremely Sparse Supervision Incentivizes Reasoning Ability

A new arXiv paper (2609.04565v1) reports that large language models can be effectively trained for reasoning with as few as one or two tokens per trajectory, just 0.05% of all generated tokens, challenging the assumption that post-training must be token-intensive. The finding, observed across nine teacher–student configurations using the Qwen3 family on mathematical reasoning and validated on coding, Llama models, and PPO-based RLVR, suggests that extremely sparse supervision can match or surpass full-token training in improving reasoning ability.

arXiv:2609.04565v1 Announce Type: new Abstract: Large language models demonstrate increasingly strong reasoning capabilities through effective post-training. Yet, prevailing post-training methods optimize over massive numbers of tokens, implicitly assuming that effective learning must be token-intensive. We revisit this assumption in the on-policy distillation OPD setting, which naturally admits dense teacher supervision at every generated token. Using the Qwen3 family, we discover a counter-intuitive phenomenon: reasoning can be effectively incentivized by an extremely small fraction of generated tokens--as few as one or two tokens per reasoning trajectory, corresponding to only 0.05% of all tokens. Surprisingly, this sparse supervision in most cases matches or surpasses full-token training in improving reasoning ability, despite excluding the vast majority of generated tokens from the training objective. This phenomenon is consistently observed across nine teacher--student configurations spanning different model scales on mathematical reasoning tasks, and is further validated on coding reasoning, Llama models and Proximal Policy Optimization PPO -based reinforcement learning with verifiable reward RLVR . Interestingly, such extremely sparse supervision may be closer to the natural learning process: rather than correcting every step word by word, one reflects on a few critical reasoning steps, updates prior understanding, and continues the trial-and-error, avoiding micro-level corrections while remaining remarkably effective. Overall, our results challenge the assumption that effective post-training must be token-intensive and point to a new direction for understanding and designing more efficient post-training algorithms.