CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning

A new arXiv paper (2610.36437v1) proposes CONCUR, an advantage estimator for multi-task, multi-reward reinforcement learning that replaces the standardized scores of each gated auxiliary reward with the closest scores satisfying two conditions: responses with higher primary rewards cannot receive lower scores, and responses at the primary's ceiling receive equal scores. The paper introduces two variants, CONCUR-T, which applies both conditions throughout training, and CONCUR-A, which lifts the second condition for an auxiliary that agrees with the primary's order more often than it reverses it among passing responses. Across three models from two families and on math, code, instruction following, and writing, both variants scored higher on average than all baselines and than dropping all gated auxiliaries.

arXiv:2610.36437v1 Announce Type: new Abstract: When a Large Language Model LLM is trained with Reinforcement Learning RL on multiple tasks, a task's primary reward can gate auxiliary rewards such as a length reward: on a failing response, an auxiliary cannot exceed the primary. However, group-relative advantage estimators treat a gated auxiliary like any other component, so it can reverse the primary's order, and at the primary's ceiling only auxiliaries order responses. Keeping gated auxiliaries hurts some tasks, while dropping them hurts others. We propose CONCUR, an advantage estimator that replaces the standardized scores of each gated auxiliary with the closest scores that satisfy two conditions: responses with higher primary rewards cannot receive lower scores, and responses at the primary's ceiling receive equal scores. CONCUR-T applies both conditions throughout training, and CONCUR-A lifts the second for an auxiliary that agrees with the primary's order more often than it reverses it among passing responses. On three models from two families and across math, code, instruction following, and writing, both variants score higher on average than all baselines and than when all the gated auxiliaries are dropped.