LLM Judge Validation Under Sparse Overlap: From Inference to Design

A new arXiv paper (2609.31857) finds that at 5% pairwise human-label overlap, LLM-judge validation can make wrong deployment decisions 25% of the time and select the wrong best judge among ten candidates 65% of the time. The paper recommends budgeting for roughly 25% overlap for non-borderline judges and allocating overlap stratified by informative slices, which can halve false rejections versus random sampling.

Agents & Inference arXiv https://arxiv.org/abs/2609.31857 LLM Judge Validation Under Sparse Overlap: From Inference to Design Which summary reads better? Pick one — models revealed after.Both summaries are AI-generated. Summary A Mistral Large quota or rate limit — check usage and plan. Original headline: LLM Judge Validation Under Sparse Overlap: From Inference to Design Summary B At 5% pairwise human-label overlap, LLM-judge validation can make wrong deployment decisions 25% of the time and pick the wrong best judge among ten candidates 65% of the time. If you ship evaluator models, sparse overlap is not a bookkeeping detail: budget for roughly 25% overlap for non-borderline judges and allocate overlap stratified by informative slices, because that can halve false rejections versus random sampling. 0 picks