{"slug": "reward-granularity-in-rlvr-comparing-process-and-outcome-reward-structures-for", "title": "Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models", "summary": "Researchers compared process and outcome reward structures for reinforcement learning with verifiable rewards (RLVR) in small language models, finding that process-only supervision achieved 63.73% accuracy on GSM8K versus 53.75% for outcome-only, a nearly 10-percentage point gap. The study, using Qwen2.5-0.5B fine-tuned with GRPO, demonstrates that reward granularity is a critical design decision for improving mathematical reasoning in small models.", "body_md": "arXiv:2607.02869v1 Announce Type: new\nAbstract: Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a promising paradigm for improving mathematical reasoning in language models. Yet most RLVR work rewards only the final answer (outcome-based rewards), leaving the impact of step-level process supervision (process rewards) underexplored especially for small models that lack the capacity to self-correct under sparse feedback. We systematically compare five reward conditions applied to Qwen2.5-0.5B fine-tuned with Group Relative Policy Optimization (GRPO) on GSM8K: a no-RL baseline, process-only, outcome-only, and three hybrid weightings ($\\lambda \\in \\{0.9, 0.5, 0.1\\}$ process weight). Process-only supervision achieves 63.73% test accuracy versus 53.75% for outcome-only, a nearly 10-percentage point gap while yielding reasoning traces with higher step validity and lower deviation from ground-truth chain length. Hybrid rewards generally correlate positively with process weight, with one notable anomaly: the low-process / high-outcome configuration ($\\lambda=0.1$) underperforms pure outcome supervision, suggesting conflicting optimization signals. Error analysis using GPT-4o as a judge reveals distinct failure mode distributions: process models generate structurally inconsistent but arithmetically grounded traces, while outcome models produce concise but derivation-error-prone chains. Our results demonstrate that reward granularity is a first-order design decision for RLVR, with process-level supervision substantially improving both accuracy and trace fidelity in small language models.", "url": "https://wpnews.pro/news/reward-granularity-in-rlvr-comparing-process-and-outcome-reward-structures-for", "canonical_source": "https://arxiv.org/abs/2607.02869", "published_at": "2026-07-07 04:00:00+00:00", "updated_at": "2026-07-07 04:12:05.796891+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research"], "entities": ["Qwen2.5-0.5B", "GSM8K", "GRPO", "GPT-4o"], "alternates": {"html": "https://wpnews.pro/news/reward-granularity-in-rlvr-comparing-process-and-outcome-reward-structures-for", "markdown": "https://wpnews.pro/news/reward-granularity-in-rlvr-comparing-process-and-outcome-reward-structures-for.md", "text": "https://wpnews.pro/news/reward-granularity-in-rlvr-comparing-process-and-outcome-reward-structures-for.txt", "jsonld": "https://wpnews.pro/news/reward-granularity-in-rlvr-comparing-process-and-outcome-reward-structures-for.jsonld"}}