Cohen's kappa

A new guide breaks down evaluation alignment between human experts and LLM judges into three measurable questions: human reliability on the rubric, LLM-human agreement relative to human-human agreemen…

A systematic evaluation of 21 LLM-as-a-Judge models across 118 runs and 541,000 judgments reveals that exact-match agreement overstates discriminative ability, with kappa deflation of 33–41 percentage…