A developer building a production AI agent in Spring Boot found that a system prompt change based on intuition lost in a pairwise A/B test against the old prompt, despite appearing better in manual ch…
LLM-as-a-Judge is a technique that uses a large language model to evaluate the output of another model, offering a scalable and explainable proxy for human preference. The approach was formalized in a…
A benchmark score is a manufactured number that passes through a chain of choices—sampling, prompting, scoring, and aggregation—each of which can change the final result while model weights stay fixed…
A developer found that improving the rubric for a small LLM judge (Qwen2.5-1.5B) did not increase its agreement with human votes, which remained around 43%. However, swapping to a larger model (DeepSe…
A new stereological theory reveals that current LLM benchmarks suffer from a structural blind spot exceeding the runner-up score gap by two orders of magnitude and dominating statistical noise by 52-1…