Reinforcement Learning with Decomposed Subtasks

A new arXiv paper (arXiv:2609.27035v1) introduces Reinforcement Learning with Decomposed Subtasks (RLDS), a method that replaces the scalar GRPO advantage with Subtask-Decomposed Advantage Estimation (SDAE), splitting trajectory reward into per-subtask shares on a fixed taxonomy before the policy update. Evaluated on four agentic benchmarks, RLDS gained 11.5 points on ScienceWorld (paired-bootstrap 95% CI [+9.8, +13.3]) and 9.8 points on FrozenLake ([+7.0, +12.8]), with gains within noise on HotpotQA and DeepResearch, and cut wall-clock time per step on ScienceWorld by 10.9% versus scalar GRPO.

arXiv:2609.27035v1 Announce Type: new Abstract: Group Relative Policy Optimization GRPO and related policy-gradient methods for training language model agents collapse an entire multi-turn rollout into a single scalar trajectory reward before it enters the policy update. When the task composes distinct skills, especially under sparse and delayed environmental feedback, this collapsing is lossy: the optimizer must implicitly infer which competency drove the outcome and how that should change behavior. We argue the right primitive is not a better scalar but a decomposition: trajectory reward should be split along subtasks before it enters the policy update. We introduce Reinforcement Learning with Decomposed Subtasks RLDS , whose core is Subtask-Decomposed Advantage Estimation SDAE : a replacement for the scalar GRPO advantage that splits trajectory reward into per-subtask shares on a fixed taxonomy, computes a group-relative advantage per subtask, and distributes per-token credit by weighting each subtask's advantage by its importance, concentrating it around the step where a reflection marks that subtask's execution as consequential. We evaluate on four agentic benchmarks: FrozenLake sparse grid navigation , HotpotQA multi-hop QA, one retrieval tool , ScienceWorld long-horizon embodied science , and DeepResearch long-form research, four tools, composite rubric reward . Heterogeneity diagnostics emitted during training show where decomposition pays off - gains scale with subtask heterogeneity, largest on the high-heterogeneity tasks ScienceWorld +11.5 points, paired-bootstrap 95% CI +9.8, +13.3 and FrozenLake +9.8 points, +7.0, +12.8 , and within noise on HotpotQA and DeepResearch, where the diagnostics predicted little to recover. ScienceWorld is also more compute-efficient under RLDS than scalar GRPO -10.9% wall-clock per step , as long rollouts amortize the fixed reflect-and-grade overhead.