Pairwise judging of saturated benchmark transcripts reduces prediction error 5.3 times
Which summary reads better? Pick one — models revealed after.Both summaries are AI-generated.
Even on benchmarks where every frontier model scores near-perfect, the solution transcripts still discriminate capability: pairwise judging of correct-answer transcripts cut capability-prediction error 5.3x versus relying on the saturated accuracy numbers, with the signal concentrated in reasoning traces and tool-call behavior. Practically, you can keep squeezing differentiating signal out of eval suites you've already "beaten" by judging how models solved tasks rather than whether they did—but beware it generalizes poorly across distribution shifts (it underestimated the first 2025 reasoning models), so don't trust transcript-based scores blind when a genuinely new model class lands.
Pairwise judging of correctly solved benchmark transcripts cut capability-prediction error by 5.3x versus using saturated scores alone. For production evals, keep full task transcripts—especially reasoning and tool-call traces—because “pass/fail” scores throw away useful model-differentiating signal, but don’t treat this as a drop-in fix for frontier model gating: the method can fail under distribution shift, including underestimating new reasoning-model classes.