Multi-Reward Reinforcement Learning for LLM Agents: Comparing PPO, GRPO, DAPO, and GDPO

A technical analysis of multi-reward reinforcement learning for LLM agents finds that summing multiple reward channels into standard algorithms like PPO or vanilla GRPO causes scale dominance, where the loudest reward channel swallows subtle ones and up to a third of expensive GPU batches generate zero gradients. The piece traces the evolution from PPO, which requires a second 27B critic model and twice as many H100s, through GRPO's critic-free group normalization, to DAPO's dynamic sampling, which discards zero-variance dead groups that can account for 30% to 40% of training steps on hard tasks. The analysis covers GDPO as the fourth approach for balancing competing rewards such as task accuracy, execution efficiency, and hard guardrails.

If you only ever train language models on toy math puzzles, reinforcement learning feels simple: did the model output 42? If yes, reward is 1. If no, reward is 0. The moment you try to train an autonomous agent for real-world enterprise work, however, a single scalar reward is an absolute illusion . In production, your agent has to juggle multiple competing, messy, non-commensurate priorities at the same time: - The Main Mission R₁ : Did it actually solve the customer’s request? e.g., execute the right SQL query, compile the circuit, return the right data payload . - Execution Efficiency R₂ : Did it solve it elegantly in 3 tool calls, or did it run a wild 40-step loop that burned $4 in API tokens and spiked database CPU? - Hard Guardrails & Constraints R₃ : Did it stay inside the sandbox? Did it strictly adhere to output JSON schemas, avoid mutating production tables, and preserve security invariants? Here is the dirty secret of post-training: if you take these three scores and simply add them together into standard algorithms like PPO or vanilla GRPO, your training run will almost certainly tear itself apart . The loudest reward channel swallows the subtle ones, the agent learns to game the system, and up to a third of your expensive GPU batches end up generating zero gradients. 1. The Algorithmic Evolution: PPO → GRPO → DAPO → GDPO To understand why modern multi-reward agent post-training looks the way it does, we must trace how policy gradient estimators evolved: 1.1 PPO: The VRAM-Hungry Workhorse Proximal Policy Optimization Schulman et al., 2017 was the engine behind the original RLHF revolution. It uses an Actor-Critic architecture: the Actor generates the tokens, and a separate Critic network learns to predict the expected future reward from state s . Why PPO Hurts in Practice: 1. The VRAM Double-Tax: If your policy is a 27B model, your Critic is usually another 27B model. You have to hold two massive models in GPU memory along with their optimizer states and activations. You end up needing twice as many H100s just to keep the critic alive. 2. Critic Drift on Multi-Reward Tasks: Trying to train a single critic head to predict a composite stew of task accuracy, latency penalties, and format compliance is notoriously unstable. The critic gets confused, advantages get noisy, and policy updates turn sluggish. 1.2 GRPO: Ditching the Critic Entirely DeepSeekMath 2024 introduced Group Relative Policy Optimization GRPO , and it felt like a breath of fresh air. GRPO tossed the Critic network into the recycling bin. Instead of asking a neural net to predict a baseline, GRPO samples a group of G candidate completions{o₁, o₂, ..., o G} for the same prompt, scores them all, and normalizes advantages against the group’s own mean and standard deviation: Instant win: GPU memory needs dropped in half But when applied to multi-objective environments, vanilla GRPO made an innocent-looking mathematical assumption called Sum-then-Normalize : As we will demonstrate below, this single line of math creates a devastating failure mode: Scale Dominance . 1.3 DAPO: Rescuing Dead Groups with Dynamic Sampling When you run GRPO on difficult engineering problems, you quickly discover the curse of Dead Groups . If a coding task is tough and all 8 candidate rollouts in a group fail with a syntax error, every single rollout gets a reward of 0. When all rewards are 0, the group standard deviation is 0. That means the advantage is 0 across the entire group Your expensive GPU cluster just spent 30 seconds generating tokens, and the gradient update is completely empty. In hard tasks, 30% to 40% of all training steps can be dead groups . DAPO introduced dynamic sampling : during rollout scoring, if a group has zero variance, it immediately discards the dead data and pulls fresh active prompts until every training batch contains real learning signal, cutting wasted GPU cycles to near zero. 1.4 GDPO: Decoupled Normalization Normalize-then-Sum Introduced in 2026 arXiv:2601.05242 and integrated into modern libraries like TRL 1.7, GDPO fixes the fundamental multi-reward flaw of GRPO. Instead of adding raw scores together and then normalizing, GDPO enforces Normalize-then-Sum : Every reward channel is normalized independently across the group first. Now, whether a reward channel naturally varies between 0, 1.0 or between 0, 0.05 , both channels have a mean of 0 and a variance of 1. The small metric can no longer be bullied by the large one. 2. Interactive Demo: Does Changing Score Units Change the Learning Signal? Suppose attempts A and B are correct, while C and D are wrong. Each also receives an efficiency score: B is both correct and efficient. C is wrong but very fast—returning an incorrect constant can be lightning-fast. Try the interactive demonstration below: toggle between Sum, then normalize and Normalize each, then sum , and switch the efficiency scale from 0–1 to 0–100: Does changing score units change the learning signal? Try this: Switch efficiency from a 0–1 scale to a 0–100 scale. Candidate behavior and reward weights stay identical. What changed: Adding raw channels makes their numerical units matter. When efficiency is multiplied by 100, the incorrect but efficient candidate C can receive the strongest positive advantage. | One prompt, four candidates, equal channel weights. Population standard deviation; zero-variance channels contribute zero. | | | | |---|---|---|---| | Attempt | Correct | Efficiency | Advantage | |---|---|---|---| | A | 1 | 0.2 | 0.35 | | B | 1 | 0.6 | 1.27 | | C | 0 | 1.0 | -0.12 | | D | 0 | 0.4 | -1.50 | Notice what happens under joint summation: when efficiency is reported on a 0–100 scale, the incorrect but fast candidate C suddenly receives a higher positive advantage than the correct solution Under decoupled normalization, candidate B remains the standout winner regardless of unit scales. 3. The Mathematics of Scale Dominance: Why Simple Sums Fail To see why Sum-then-Normalize fails mathematically, look at the variance of a sum of two independent reward signals: In an agent task: - Channel 1 Task Success is binary: r₁ ∈ {0, 1}. Its variance is roughly σ₁² ≈ 0.25. - Channel 2 Token Efficiency is small: r₂ ∈ 0, 0.05 . Its variance is tiny: σ₂² ≈ 0.0006. When you sum them up, Channel 1 accounts for 99.7% of the total variance . When GRPO divides by σ<sub R</sub , Channel 2 is effectively multiplied by zero. The model quickly learns a toxic heuristic: “I can spam 50 unnecessary tool calls and burn huge token budgets, because the efficiency penalty is mathematically invisible to my gradients ” 4. Geometric Reward Collapse: When Winners Look Like Losers Under vanilla GRPO, distinct trade-offs get mashed into the exact same scalar score: - Candidate A: Perfect task solution R₁ = 1.0 , but horribly violates formatting rules R₃ = 0.0 . Total = 1.0. - Candidate B: Flawless format and safe execution R₃ = 0.2 , but solved 80% of the core task R₁ = 0.8 . Total = 1.0. To vanilla GRPO, both candidates look identical A<sub A</sub = A<sub B</sub . The policy receives zero gradient to choose the clean, compliant solution over the broken, unsafe one. With GDPO’s decoupled normalization, Candidate B’s excellence on the constraint channel stands out with a strong positive sub-advantage, guiding the model toward the true Pareto frontier. 5. Hard Telemetry from gft-studio We tested all four algorithms on an identical multi-objective agent gym in gft-studio , evaluating task success R₁ , token efficiency R₂ , and constraint compliance R₃ on a 27B model: | Method & Normalization | Critic Model? | Composite Pareto Reward | Task Success Rate | Constraint Violation Rate | Dead Group Rate | |---|---|---|---|---|---| | PPO Actor-Critic Baseline | Yes 27B Critic, GAE | 0.312 ± 0.04 | 42.5% | 24.8% | N/A GAE baseline | | GRPO Vanilla Joint Sum | No Critic-Free | 0.395 ± 0.03 | 51.2% | 29.4% | 34.2% Dead Groups | | DAPO Dynamic Refill + GRPO | No Critic-Free | 0.448 ± 0.02 | 56.8% | 22.1% | < 3.0% Refilled | | GDPO Decoupled Norm + Safety | No Critic-Free | 0.562 ± 0.02 | 63.4% | < 3.8% | < 3.0% Refilled | The Big Takeaways - Vanilla GRPO Bleeds Safety Constraints: While it solved the primary task 51.2% of the time, its constraint violation rate was alarming: 29.4% . Because task reward dominated the sum, the agent routinely broke formatting and safety contracts to get the job done. - DAPO Saves 34% of Wasted Compute: Over a third of vanilla GRPO groups had zero variance. DAPO’s dynamic prompt replenishment ensured that every batch drove meaningful parameter updates. - GDPO Nails the Pareto Frontier: By standardizing each reward channel independently, constraint violations collapsed from 29.4% down to under 3.8% , while overall task success jumped to 63.4% . 6. The Practitioner’s Field Guide When building reinforcement learning pipelines for real-world enterprise agents: 1. Never use joint summation for multi-reward environments. Always normalize channels independently first Normalize-then-Sum . 2. Hard constraints belong in the tool interface, not in soft weights. If an agent must never write unauthorized data to production, block the write at the API barrier. Do not rely solely on negative rewards to enforce safety. 3. Turn on dynamic group sampling DAPO if your task is hard. If your baseline model solves the problem less than 20% of the time, you will waste huge amounts of GPU compute on dead groups without it. 4. Clamp your combined advantages. When combining multiple normalized channels, an outlier rollout that spikes on two channels simultaneously can cause massive policy drift. Always apply a safety clamp c <sub max</sub ∈ 3.0, 5.0 to protect training stability.