OpenAI Evals

A developer found that an LLM judge gave opposite results for the same eval run on consecutive days due to position bias, one of three systematic biases documented in the 2023 paper "Judging LLM-as-a-…

A developer warns that 38% of AI teams cite 'evaluation debt' as their primary blocker, where offline agent evals fail to catch production failures because they measure past data rather than shifting …