{"slug": "prompt-lottery-in-llm-as-a-judge-measuring-and-mitigating-sensitivity-to", "title": "Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing", "summary": "A large-scale arXiv study (2610.38076v2) finds that paraphrastic rephrasings of judge prompts can alter LLM-as-a-Judge evaluation outcomes and, in some settings, reverse relative model rankings — a phenomenon the authors call \"Prompt Lottery.\" A user study with 10 human annotators showed LLMaJ judges suffer 6-9x larger agreement degradation under prompt rephrasings than humans, falling below the generally accepted threshold in usability studies. The authors conclude single-prompt evaluation is insufficiently robust for model ranking and show multi-prompt aggregation substantially improves robustness by emphasizing judgments consistent across prompt variants.", "body_md": "arXiv:2610.38076v2 Announce Type: replace \nAbstract: LLM-as-a-Judge has become a standard for evaluating large language models, yet most approaches implicitly assume that a single judge prompt functions as a stable evaluation instrument. Prior work has primarily explored deliberate prompt modifications such as scoring rubrics, role descriptions, or chain-of-thought reasoning in order to optimize accuracy. In contrast, we investigate whether validated prompt variants intended to preserve evaluation semantics while differing in surface realization (\\textit{paraphrastic rephrasings}) yield consistent judgments under a fixed evaluation task. We introduce \\textit{Prompt Lottery}, a phenomenon in which paraphrastic rephrasings produce different LLMaJ judgments. Across a large-scale study spanning diverse evaluation tasks, judge families, and model backbones, we find that these variations can alter evaluation outcomes and, in some settings, reverse relative model rankings. A user study with 10 human annotators shows that LLMaJ judges exhibit 6-9$\\times$ larger agreement degradation under prompt rephrasings than humans, falling below the generally accepted threshold in usability studies. This indicates that single-prompt evaluation is insufficiently robust for model ranking. Finally, we show that multi-prompt aggregation substantially improves robustness by emphasizing judgments that remain consistent across prompt variants.", "url": "https://wpnews.pro/news/prompt-lottery-in-llm-as-a-judge-measuring-and-mitigating-sensitivity-to", "canonical_source": "https://www.machinebrief.com/news/prompt-lottery-in-llm-as-a-judge-measuring-and-mitigating-se-r0y4", "published_at": "2026-11-09 05:00:00+00:00", "updated_at": "2026-11-09 06:21:33.423859+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-safety"], "entities": ["Prompt Lottery", "LLM-as-a-Judge", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/prompt-lottery-in-llm-as-a-judge-measuring-and-mitigating-sensitivity-to", "markdown": "https://wpnews.pro/news/prompt-lottery-in-llm-as-a-judge-measuring-and-mitigating-sensitivity-to.md", "text": "https://wpnews.pro/news/prompt-lottery-in-llm-as-a-judge-measuring-and-mitigating-sensitivity-to.txt", "jsonld": "https://wpnews.pro/news/prompt-lottery-in-llm-as-a-judge-measuring-and-mitigating-sensitivity-to.jsonld"}}