Chatbot Arena

A benchmark score is a manufactured number that passes through a chain of choices—sampling, prompting, scoring, and aggregation—each of which can change the final result while model weights stay fixed…

A developer found that improving the rubric for a small LLM judge (Qwen2.5-1.5B) did not increase its agreement with human votes, which remained around 43%. However, swapping to a larger model (DeepSe…

A new stereological theory reveals that current LLM benchmarks suffer from a structural blind spot exceeding the runner-up score gap by two orders of magnitude and dominating statistical noise by 52-1…