CrucibleBench finds an LLM judge shifted rankings by six spots

Removing two classifier-dependent scoring dimensions from CrucibleBench shifted language model leaderboard positions by as many as six spots, researchers Benjamin Davis and Philip Mims reported. The experiment, which ran 13 models through a text dungeon for 650 runs at a cost of $99.59, found that the primary classifier labeled 91.6% of sampled turns as objective probes versus 47.8% for a second judge, with aggregate Cohen's kappa of 0.04 indicating poor agreement.

In May, Benjamin Davis and Philip Mims published a CrucibleBench https://cruciblebench.ai/?ref=runtimewire experiment that sent 13 language models into a persistent text dungeon for 650 runs. Their most useful result came from the machinery doing the grading: removing two classifier-dependent scoring dimensions moved models by as many as six leaderboard positions. In a launch post preserved by an RSS mirror https://hacklogg.com/p/%21aNiU0b?ref=runtimewire , the researchers said the technical report emerged from several months of nights-and-weekends work on personal computers. CrucibleBench spent $99.59 on the retained, scored experiment, giving Davis and Mims a cheap way to test a larger idea: whether an old multi-user dungeon, or MUD, can expose behavioral failures that disappear inside static question-and-answer benchmarks. Public biographical detail on the researchers is limited. CrucibleBench's about page https://cruciblebench.ai/about/?ref=runtimewire identifies Davis as a co-author and the maintainer of the implementation and public research materials, while Mims is listed as the other Phase 1 co-author. The site describes CrucibleBench as an independent applied research effort operated by Folcright LLC, rather than a venture-backed evaluation startup. That small-scale origin shaped the design. Davis and Mims used mature text-game mechanics instead of building a photorealistic simulator, browser environment, or elaborate virtual office. The compact world contains 12 rooms, four non-player characters, 14 items, seven command types, and two hidden social objectives. Each run gives a model 50 turns. NPC trust and suspicion scores move from zero to 100 as the model talks, explores, gives away items, and pursues its assigned objective. The judge became the experiment CrucibleBench scored models across Goal Pursuit, Social Adaptation, World Grounding, and Strategic Sophistication. World Grounding was deterministic, while Goal Pursuit and Strategic Sophistication relied heavily on labels produced by Google Gemini 3.1 Flash Lite at a temperature setting of 0.0 https://cruciblebench.ai/whitepaper.pdf?ref=runtimewire . No human rater scored the original outputs, according to the report. Davis and Mims compared the classifier-dependent scoring with a second judge. Per-model agreement ranged from 21.7% to 84.8%, while aggregate Cohen's kappa was 0.04. The whitepaper https://cruciblebench.ai/whitepaper.pdf?ref=runtimewire identifies the second judge as Claude Sonnet 4.6 and says the audit covered 810 sampled dialogue turns. Agreement on objective-probe labels ranged from 21.7% for Gemini 3.1 Pro to 84.8% for Qwen 3.5 397B. The ranking consequences were substantial. Under the full composite, GPT-5.4 ranked first and Gemini 3.1 Pro ranked third. Under a classifier-minimized subtotal, GPT-5.4 fell to fifth and Gemini dropped to ninth. Claude Sonnet 4.6 moved from fourth to first, while DeepSeek R1 rose from seventh to second. Six shifts remained outside the paper's scenario-sampling noise under its 90% paired block-bootstrap procedure. The authors handle that result carefully. Removing two dimensions changes the behavior being measured, so movement alone cannot prove that the original leaderboard was distorted. The sharper warning comes from the judge audit: the primary classifier labeled 91.6% of sampled turns as objective probes, compared with 47.8% for Claude Sonnet 4.6 https://cruciblebench.ai/whitepaper.pdf?ref=runtimewire , and disagreement varied sharply by evaluated model. Gemini 3.1 Pro, the largest ranking mover, belonged to the same model family as the primary classifier. Davis and Mims treat family bias as an open hypothesis because Claude Sonnet 4.6 was also among the models evaluated. That caveat strengthens the work. CrucibleBench does not present its table as a definitive model ranking. Confidence intervals overlap among much of the frontier group, the experiment lacks a human baseline, and the paper says the prototype has not been validated as a measure of general social intelligence or real-world agent performance. The leaderboard is evidence about the instrument, with the instrument's own instability becoming the main finding. What a text dungeon makes visible A MUD gives the researchers a finite action space and a complete transcript. The world can tell when a model tries to speak to an NPC who is in another room, repeatedly examines its surroundings without acting, or gets stuck using the same conversational strategy. Items remain where a model leaves them, and trust accumulated earlier in a run continues to shape later interactions. Those constraints produced readable failures. Dialogue looping appeared in 14% to 66% of frontier-model runs and was the dominant flagged failure mode for every frontier model tested. Grok 4 attempted meaningful numbers of wrong-room interactions in 12% of its runs. The open OLMo 3.1 32B floor model tried to address guards who were absent, attempted to converse with an item called a street crystal, and spent its final 15 turns repeating interactions with the same captain. The transcripts also show successful behavior in terms that a developer can inspect. In one run highlighted by CrucibleBench https://cruciblebench.ai/?ref=runtimewire , GPT-5.4 navigated to a barracks, found a signet ring, returned it to the likely owner, and used that gesture to earn enough trust for a recommendation. It completed the objective on turn 14 of a 50-turn allowance. That kind of trace is easier to debug than a composite score whose components depend on an opaque judge. The whitepaper reports $99.59 in billed Phase 1 API usage, and CrucibleBench released the billing export alongside the transcripts, source and scoring code. Its accounting https://cruciblebench.ai/whitepaper.pdf?ref=runtimewire splits that retained experiment into $98.85 for primary model calls and $0.74 for classifier calls. Including discarded calibration attempts, all API expenditure totaled $129.14. Appendix E says the figures derive from a 43,372-row OpenRouter activity-log export dated March 11, 2026. Cost also failed to predict behavioral quality across the 12 frontier models. Grok 4 generated $41.71 in primary-model charges, or 42.2% of the $98.85 spent on primary model calls, according to the whitepaper's cost accounting https://cruciblebench.ai/whitepaper.pdf?ref=runtimewire . It scored below the median under both scoring configurations. The authors report https://cruciblebench.ai/research/?ref=runtimewire no significant relationship between log inference cost and behavioral score, though the sample is too small to support a broad claim about model economics. A tiny project enters a well-funded eval market Braintrust, an AI evaluation platform, raised an $80 million Series B in February https://www.braintrust.dev/blog/announcing-series-b?ref=runtimewire . CrucibleBench is pursuing a narrower research question with a radically smaller cash budget: can a constrained persistent world produce behavioral evidence that is cheap, replayable, and understandable? The timing also follows warnings from the model labs. Anthropic's January guidance https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents?ref=runtimewire recommends deterministic graders where possible and says model-based graders require calibration with humans. On July 8, OpenAI reported https://openai.com/index/separating-signal-from-noise-coding-evaluations/?ref=runtimewire that roughly 30% of tasks in SWE-Bench Pro appeared broken, another example of evaluation design overwhelming the model signal a benchmark was supposed to measure. Davis and Mims are planning a larger Phase 2 https://cruciblebench.ai/phase-2/?ref=runtimewire with human baselines, multiple classifiers, preregistered adaptation rules, and a hidden-evaluation split. The current instrument-validation plan calls for roughly 1,370 to 1,450 model runs. CrucibleBench lists a provisional $3,500 fundraising envelope, while the live plan https://cruciblebench.ai/phase-2-budget.xlsx?ref=runtimewire gives low and high budget scenarios of about $1,078 to $8,448 when an optional expansion is included. Engineering and analysis remain contributed in kind. Phase 2 is still being built https://cruciblebench.ai/phase-2/?ref=runtimewire . The project says its single-run harness has passed a live-model smoke test, while the publishable environment, objective bindings, calibration, and preregistration remain unfinished. CrucibleBench is recruiting funding partners, technical collaborators, and post-calibration pilot participants. The Phase 2 page also says the project will not sell composite scores or leaderboard placement before validation. That restraint is the project's strongest commercial instinct. Davis and Mims have produced a small research artifact that surfaces concrete agent failures and a larger warning about the evaluation business: a benchmark can generate precise numbers long before it has earned confidence in what those numbers mean.