# Blind Spots Travel, Knowledge Stays Home: What RL Inherits from an LLM Judge

> Source: <https://arxiv.org/abs/2611.25191>
> Published: 2026-11-20 05:00:00+00:00

**Recommendation:** Audit judge verdicts at the category level, not just overall accuracy: hand-checking 12 verdicts shifted refusal from 5% to 90%, and self-consistency voting over 32 untrained samples matched the answer key without a judge.
More in the playbook "Evals and testing for LLM apps": <https://wpnews.pro/playbooks/evals-testing>

arXiv:2611.25191v1 Announce Type: new 
Abstract: LLM judges serve as rewards where no answer key exists, in the hope that their knowledge passes to the trained model. In 814 RL runs that teach 7-8B models to answer closed-book questions or abstain, we ask what passes. What a grader knows about single answers stays home. A wrong answer that a judge rewards is memorised for that one question. A model trained with the answer key, or with a far stronger judge, chooses which new questions to answer no better than the untrained model does by sampling 32 answers and abstaining where they disagree; the grader's knowledge adds about one corrected answer per hundred questions. What a grader believes about a kind of question travels. A judge that rejects eleven of the twelve correct training answers about capitals of former states teaches the model to refuse 98% of such capitals on new questions, a loss that accuracy and abstention rates do not reveal. Changing only those twelve verdicts by hand moves the refusal from 5% to 90%, even for a grader that accepts as many answers as the answer key; at that leniency, two of four judges already cost 38-42% of these capitals. The capitals are not forgotten: with "I don't know" blocked, the model answers 94% of them correctly. The grader therefore needs no knowledge of its own: rewarding the model for giving the answer that most of its own untrained samples agree on, with no labels and no judge, earns the answer key's reward and keeps the category.
