arXiv:2611.25191v1 Announce Type: new Abstract: LLM judges serve as rewards where no answer key exists, in the hope that their knowledge passes to the trained model. In 814 RL runs that teach 7-8B models to answer closed-book questions or abstain, we ask what passes. What a grader knows about single answers stays home. A wrong answer that a judge rewards is memorised for that one question. A model trained with the answer key, or with a far stronger judge, chooses which new questions to answer no better than the untrained model does by sampling 32 answers and abstaining where they disagree; the grader's knowledge adds about one corrected answer per hundred questions. What a grader believes about a kind of question travels. A judge that rejects eleven of the twelve correct training answers about capitals of former states teaches the model to refuse 98% of such capitals on new questions, a loss that accuracy and abstention rates do not reveal. Changing only those twelve verdicts by hand moves the refusal from 5% to 90%, even for a grader that accepts as many answers as the answer key; at that leniency, two of four judges already cost 38-42% of these capitals. The capitals are not forgotten: with "I don't know" blocked, the model answers 94% of them correctly. The grader therefore needs no knowledge of its own: rewarding the model for giving the answer that most of its own untrained samples agree on, with no labels and no judge, earns the answer key's reward and keeps the category.
Blind Spots Travel, Knowledge Stays Home: What RL Inherits from an LLM Judge
In 814 reinforcement learning runs teaching 7-8B models to answer closed-book questions or abstain, an LLM judge's knowledge of individual answers did not transfer to the trained model, according to arXiv paper 2611.25191v1. A judge that rejected eleven of twelve correct training answers about capitals of former states taught the model to refuse 98% of such capitals on new questions, and hand-changing only those twelve verdicts moved the refusal rate from 5% to 90%. The authors report that rewarding the answer most of the model's own untrained samples agree on, with no labels and no judge, earns the answer key's reward and preserves the category.
Audit judge verdicts at the category level, not just overall accuracy: hand-checking 12 verdicts shifted refusal from 5% to 90%, and self-consistency voting over 32 untrained samples matched the answer key without a judge.
more in playbook: Evals and testing for LLM appsRun your AI side-project on zahid.host
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.