arXiv:2611.25135v1 Announce Type: cross Abstract: Reinforcement learning (RL) can only teach a language model what its reward can recognize. Answer checkers are exact but need a reference answer; learned reward models and LLM judges work everywhere, but their scores are not probabilities. We study RLCDR (Reinforcement Learning from Calibrated Decision Rewards). An LLM converted into a decision model answers a typed question about a response ("is the final answer correct?", "which step is the first error?") with a probability distribution, read in one forward pass, debiased over option orders and calibrated with about a hundred labels. Training Qwen3-1.7B with GRPO on MATH, this reference-free reward raises MATH500 accuracy from 72.2% to 76.8% (+4.65 points, p < 1e-4), matching training on the answer key (76.9%), while a random reward gains only +0.65. The gain grows with difficulty, from 44.8% to 51.7% on the hardest level. On the policy's own changing samples the reward stays calibrated: its expected calibration error is half that of a softmax judge on the same 27B backbone (0.074 vs. 0.141), and it shows no sign of reward hacking. Without step training, the same model finds the first wrong step better than a dedicated 7B process reward model (ProcessBench F1 0.772 vs. 0.722, 20/20 splits). We also map where the approach stops being enough: trained only on problems the base model mostly fails, the reference-free reward still improves the policy (+1.9 MATH500 points, p = 0.03) but trails the answer key, because its probabilities become overconfident on that distribution. We state six hypotheses in advance and test each with a control designed to falsify it.
Calibrated Decision Rewards: Training Language Models with a Judge That Knows How Sure It Is
Training Qwen3-1.7B with GRPO on MATH using RLCDR (Reinforcement Learning from Calibrated Decision Rewards), a reference-free reward built from a calibrated LLM decision model, raised MATH500 accuracy from 72.2% to 76.8% (+4.65 points, p < 1e-4), according to the arXiv paper 2611.25135v1. The gain nearly matched training on the answer key (76.9%), while a random reward added only +0.65 points, and the improvement grew with difficulty, from 44.8% to 51.7% on the hardest level. The calibrated reward's expected calibration error was 0.074 versus 0.141 for a softmax judge on the same 27B backbone, and without step training the model identified the first wrong step better than a dedicated 7B process reward model (ProcessBench F1 0.772 vs. 0.722, 20/20 splits).
Calibrate a typed-question LLM judge with roughly 100 labels and debias it over option orders before using it as an RL reward; it matched answer-key training on MATH500 (76.8% vs. 76.9%) but became overconfident on problems the base model mostly fails.
more in playbook: Evals and testing for LLM appsRun your AI side-project on zahid.host
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.