RewardBench

Researchers introduce program distillation as a cheaper, faster, and more transparent alternative to LLM-as-a-judge for automated evaluation, presenting PAJAMA, a system that synthesizes programmatic …

A new study finds that panels of nine large language model judges effectively provide only about two independent votes' worth of information, because the models make correlated errors on the same item…

A systematic evaluation of 21 LLM-as-a-Judge models across 118 runs and 541,000 judgments reveals that exact-match agreement overstates discriminative ability, with kappa deflation of 33–41 percentage…