RLVR works beautifully for math. Generate a response, extract the final answer, compare it to ground truth, and assign 1 or 0. No reward model to train, no human preference data to collect, no reward hacking against a learned proxy, just a deterministic check. This is why DeepSeek-R1-Zero could go from 15.6% to 77.9% on AIME with nothing but GRPO and a verifier.
Now try to apply the same idea to an agent that searches the web, calls three APIs, reads the results, and books a flight. What’s the verifier? “Did it book the right flight” is a valid terminal reward, but it’s the only signal across a 15-step trajectory, and most of those steps had nothing to do with correctness in a way you can check with a string comparison.
This is where most teams doing agentic RL get stuck. The RLVR playbook that worked for math and code doesn’t transfer cleanly to tool-using agents, and the honest answer is that reward design here is closer to systems engineering than to machine learning.
TL;DR
RLVR is the standard method for improving LLM reasoning and has shown strong results in verifiable domains like math and competitive programming, but its effectiveness drops sharply in agentic environments. The reason is structural, not incidental.
In a math problem, the trajectory is short (a few hundred tokens of reasoning) and the final answer fully determines correctness. In a tool-using agent trajectory, you might have 10–20 turns, each involving a tool call, an observation, and a reasoning step. A single terminal reward has to somehow propagate credit back across all of those turns — and with GRPO-style group-relative advantages, if every rollout in a group fails the same way, you get zero gradient signal despite non-zero variation in how they failed.
This is the sparse reward problem, and it’s the reason RLVR training in multi-turn tool-interaction settings suffers from reward hacking when using a purely outcome-based reward — the policy finds degenerate shortcuts because the reward surface gives it no reason not to.
The most direct fix is decomposing the terminal reward into intermediate, still-verifiable signals. For tool-calling agents, verifiable turn-level rewards can be constructed from tool execution success and evidence of progress toward the answer. Concretely:
A tool execution reward awards partial credit when the tool call is correctly formatted, and the environment doesn’t return an error. A separate reward checks whether the correct answer appears anywhere in the tool’s returned results, not whether the agent has already synthesized the final answer, just whether the retrieval step surfaced the right information.
def turn_level_reward(turn: dict, ground_truth_answer: str) -> float: reward = 0.0 # Tool execution reward: was the tool called correctly? if turn.get("tool_call_well_formed") and not turn.get("tool_error"): reward += 0.2 # Progress reward: does the tool's output contain the answer? tool_output = turn.get("tool_result", "") if ground_truth_answer.lower() in tool_output.lower(): reward += 0.5 return rewarddef trajectory_reward(trajectory: list[dict], ground_truth: str) -> float: # Sum turn-level rewards, then add the outcome reward at the end turn_rewards = sum( turn_level_reward(t, ground_truth) for t in trajectory ) final_answer = trajectory[-1]["content"] outcome_reward = 1.0 if ground_truth.lower() in final_answer.lower() else 0.0 return turn_rewards + outcome_reward
This gives the policy gradient signal at every step where the agent did something checkable, instead of only at the end. It’s a meaningful improvement over outcome-only rewards, but it only covers what you can verify with string matching or execution success — which is a fraction of what actually matters in a good trajectory (did the agent choose an efficient path, did it avoid redundant tool calls, did it reason coherently between steps).
For behaviors that can’t be checked with a rule, the alternative is a learned process reward model (PRM) that scores intermediate steps. Process Reward Models automatically extract intermediate milestones from demonstrations to provide dense progress estimations, addressing the long-horizon sparsity problem without requiring every step to be independently verifiable by a rule.
The tradeoff is real: a PRM is a trained model, which means it can be gamed, it requires its own training data (usually derived from expert demonstrations or successful trajectories), and it adds inference cost to your RL loop since you’re scoring every intermediate step during rollout collection.
An alternative to a trained PRM is a rubric-based LLM judge — scoring each predicted step against a structured, multi-dimensional rubric rather than a single scalar, which reduces the judge’s tendency to collapse a complex behavior into a single noisy number. If the judge fails to produce a valid score, the safe default is to zero out that reward rather than guess — a silent bad score is worse than a visibly failed one you can filter out downstream.
RUBRIC_DIMENSIONS = [ "tool_choice_appropriateness", "argument_correctness", "result_interpretation", "progress_toward_goal", "reasoning_coherence",]def llm_judge_reward(step: dict, context: dict, judge_client) -> float: """ Score a single agent step against a structured rubric. Returns a normalized [0, 1] reward, or 0.0 if the judge fails to produce a parseable score (fail-safe, not fail-silent). """ scores = [] for dimension in RUBRIC_DIMENSIONS: response = judge_client.score( step=step, context=context, dimension=dimension, scale="1-5", ) parsed = try_parse_score(response) if parsed is None: return 0.0 # fail-safe default, logged for offline review scores.append(parsed) return sum(scores) / (len(scores) * 5) # normalize to [0, 1]
In math RLVR, reward hacking looks like format exploitation — the model learns to produce output that pattern-matches the expected answer format without actually solving the problem. In agentic RLVR, the exploits are structurally different and harder to catch:
Tool-call theater: the agent learns to call tools because tool-calling is rewarded, without ever using the returned information in its final answer. Your turn-level reward for “tool executed successfully” is satisfied; the agent has learned nothing about actually using tools.
Retry gaming: if your environment allows retries and your reward only checks the final state, the policy learns to retry blindly until something passes, rather than reasoning about why a previous attempt failed.
Judge gaming: if you’re using an LLM judge for non-verifiable dimensions, the policy can learn surface patterns that the judge rewards — verbose reasoning that sounds thorough without being correct, or hedged language that scores well on “reasoning coherence” without committing to a checkable claim.
The fix for all three is the same discipline: audit your reward function the way you’d audit a metric that’s about to become a target. Ask “what’s the cheapest way to maximize this reward without doing the task” before you start training, not after you notice the model doing it.
def audit_reward_function(reward_fn, sample_trajectories: list[dict]) -> dict: """ Run cheap, degenerate trajectories against your reward function before training. If any of these score highly, your reward has a hole. """ adversarial_probes = { "empty_reasoning_valid_tool_call": ..., # tool call w/ no reasoning "tool_call_ignored_output": ..., # calls tool, ignores result "verbose_but_wrong": ..., # long reasoning, wrong answer "minimal_effort_lucky_guess": ..., # skips steps, guesses right } return { name: reward_fn(probe) for name, probe in adversarial_probes.items() } # Any of these scoring above ~0.5 signals an exploitable reward shape.
A more structural fix than reward decomposition alone is injecting guidance during training rather than relying purely on reward shape. Agent-RLVR incorporates agent guidance through a multi-agent framework where teacher guidance provides targeted information about failing patterns and expectations during RL training, functioning similarly to how a junior engineer gets pointed in the right direction by a senior teammate rather than being left to find the fix through unguided trial and error.
This matters because agentic RL settings characterized by multi-step, complex problem-solving lead to high failure rates even for frontier models, since the reward landscape is too sparse for effective training via conventional RLVR alone. No amount of reward decomposition fixes a setting where the model simply never stumbles into a successful trajectory during rollout collection. Guidance — hints, partial solutions, or corrective context injected mid-trajectory during training — increases the rate of successful rollouts without compromising the verifiability of the final reward signal, since the hints only affect exploration, not scoring.
The practical takeaway: if your agent’s task success rate during early RL rollouts is near zero, more reward engineering won’t save you. You need either an easier curriculum, guidance injection, or expert demonstration data to bootstrap from — reward shape can’t manufacture successful trajectories that never occur.
1. Dense rewards can suppress exploration just as much as sparse ones.
It’s tempting to over-engineer turn-level rewards until every plausible good behavior gets some credit. This can backfire: if partial credit is too generous, the policy converges on “safe” trajectories that collect small guaranteed rewards (call a tool, get 0.2, stop trying) rather than pushing for the larger reward that requires more exploration. Weight terminal task success meaningfully higher than intermediate process rewards — process rewards should shape behavior, not replace the incentive to actually finish the task.
2. Reward hacking gets worse, not better, as your policy improves.
A weak policy fails obviously. A policy that’s 80% of the way to competent finds subtler exploits, because it has enough capability to identify the reward function’s blind spots rather than stumbling past them. Re-run your adversarial reward audits periodically during training, not just once before you start.
3. Non-determinism in tool environments breaks reward consistency.
If your agent calls a live search API or a web tool, the same action can return different results across rollouts — meaning identical trajectories can receive different rewards. This adds noise that compounds with GRPO’s group-relative advantage computation, since your “identical” prompts aren’t producing directly comparable groups anymore. Where possible, use cached or replayed tool responses during RL training and reserve live environments for evaluation.
Reward design for tool-using agents is not a smaller version of reward design for math — it’s a different problem. Outcome-only verification, which is RLVR’s core strength in single-shot domains, becomes a liability once trajectories span multiple steps with tool calls in between. The fix isn’t one technique; it’s a layered system: turn-level verifiable rewards where you can get them, process reward models or rubric-based judges where you can’t, systematic reward auditing to catch hacking before it compounds, and guidance injection when sparsity is severe enough that no reward shape will manufacture a successful trajectory on its own.
The teams shipping real agentic RL systems in 2026 aren’t the ones with the cleverest single reward function. They’re the ones treating reward design as an evolving system with its own test suite — one that gets audited as rigorously as the policy it’s training.
Reward Design Is the Hard Part: Building Verifiable Rewards for Tool-Using Agents was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.