CrucibleBench

An independent research group's CrucibleBench experiment found that LLM-judge-based evaluation rankings are highly sensitive to classifier components, with aggregate κ on probe detection at 0.04 and p…

Removing two classifier-dependent scoring dimensions from CrucibleBench shifted language model leaderboard positions by as many as six spots, researchers Benjamin Davis and Philip Mims reported. The e…

CrucibleBench, a proof-of-concept evaluation framework that places large language models in a persistent MUD (multi-user dungeon) over 50 turns with hidden social objectives, found that a single LLM-j…