{"slug": "reinforcement-learning-with-decomposed-subtasks", "title": "Reinforcement Learning with Decomposed Subtasks", "summary": "A new arXiv paper (arXiv:2609.27035v1) introduces Reinforcement Learning with Decomposed Subtasks (RLDS), a method that replaces the scalar GRPO advantage with Subtask-Decomposed Advantage Estimation (SDAE), splitting trajectory reward into per-subtask shares on a fixed taxonomy before the policy update. Evaluated on four agentic benchmarks, RLDS gained 11.5 points on ScienceWorld (paired-bootstrap 95% CI [+9.8, +13.3]) and 9.8 points on FrozenLake ([+7.0, +12.8]), with gains within noise on HotpotQA and DeepResearch, and cut wall-clock time per step on ScienceWorld by 10.9% versus scalar GRPO.", "body_md": "arXiv:2609.27035v1 Announce Type: new \nAbstract: Group Relative Policy Optimization (GRPO) and related policy-gradient methods for training language model agents collapse an entire multi-turn rollout into a single scalar trajectory reward before it enters the policy update. When the task composes distinct skills, especially under sparse and delayed environmental feedback, this collapsing is lossy: the optimizer must implicitly infer which competency drove the outcome and how that should change behavior. We argue the right primitive is not a better scalar but a decomposition: trajectory reward should be split along subtasks before it enters the policy update. We introduce Reinforcement Learning with Decomposed Subtasks (RLDS), whose core is Subtask-Decomposed Advantage Estimation (SDAE): a replacement for the scalar GRPO advantage that splits trajectory reward into per-subtask shares on a fixed taxonomy, computes a group-relative advantage per subtask, and distributes per-token credit by weighting each subtask's advantage by its importance, concentrating it around the step where a reflection marks that subtask's execution as consequential. We evaluate on four agentic benchmarks: FrozenLake (sparse grid navigation), HotpotQA (multi-hop QA, one retrieval tool), ScienceWorld (long-horizon embodied science), and DeepResearch (long-form research, four tools, composite rubric reward). Heterogeneity diagnostics emitted during training show where decomposition pays off - gains scale with subtask heterogeneity, largest on the high-heterogeneity tasks ScienceWorld (+11.5 points, paired-bootstrap 95% CI [+9.8, +13.3]) and FrozenLake (+9.8 points, [+7.0, +12.8]), and within noise on HotpotQA and DeepResearch, where the diagnostics predicted little to recover. ScienceWorld is also more compute-efficient under RLDS than scalar GRPO (-10.9% wall-clock per step), as long rollouts amortize the fixed reflect-and-grade overhead.", "url": "https://wpnews.pro/news/reinforcement-learning-with-decomposed-subtasks", "canonical_source": "https://arxiv.org/abs/2609.27035", "published_at": "2026-09-24 04:00:00+00:00", "updated_at": "2026-09-24 04:29:59.428866+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research", "ai-agents"], "entities": ["RLDS", "SDAE", "GRPO", "ScienceWorld", "FrozenLake", "HotpotQA", "DeepResearch", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/reinforcement-learning-with-decomposed-subtasks", "markdown": "https://wpnews.pro/news/reinforcement-learning-with-decomposed-subtasks.md", "text": "https://wpnews.pro/news/reinforcement-learning-with-decomposed-subtasks.txt", "jsonld": "https://wpnews.pro/news/reinforcement-learning-with-decomposed-subtasks.jsonld"}}