{"slug": "can-a-mud-evaluate-llms-a-99-proof-of-concept", "title": "Can a MUD evaluate LLMs? A $99 proof of concept", "summary": "CrucibleBench, a proof-of-concept evaluation framework that places large language models in a persistent MUD (multi-user dungeon) over 50 turns with hidden social objectives, found that a single LLM-judge component inside its scoring stack reordered the leaderboard by up to six positions while aggregate reliability statistics remained silent. The project tested 13 models across 650 runs at a total cost of $99.59, and the authors report every result under two scoring configurations, treating the divergence as the paper's most generalizable finding.", "body_md": "# Old worlds for new agents.\n\nCrucibleBench places language models in a persistent MUD, a text world where\n**NPCs remember, trust accumulates, and mistakes leave traces**, and scores what they do\nover 50 turns with hidden social objectives.\n\n**Status:** Phase 1 proof-of-concept released · 13 models · 650 runs · $99.59 billed. Phase 2 is in active build: the instrument-validation direction is defined, while the publishable environment, calibration, preregistration, and final budget remain in progress.\n\n## Lateral thinking with withered technology\n\nNintendo's Gunpei Yokoi used the phrase to describe a design philosophy: take mature, inexpensive, well-understood technology and use it in a new way. CrucibleBench applies it to AI evaluation.\n\nInstead of photorealistic simulation or browser automation, we start with a **MUD**: a\nmulti-user dungeon, the persistent text worlds of the early internet. Its constraints are the point: a\nlimited command space makes hallucinated actions detectable, NPCs with trust and suspicion state give\nexplicit social feedback, and within-run persistence means items taken stay taken and trust earned stays\nearned.\n\nWe did not choose a MUD because it is charming. We chose it because its constraints make behavior measurable.\n\n## Old constraints solve modern measurement problems\n\nStatic benchmarks measure what models know in isolation. They do not measure how models behave where trust must be earned, information is gated by relationships, and blunt questioning raises suspicion.\n\n-\n01\n### An enumerable action space\n\n7 command types, 12 rooms, 14 items. Hallucinated actions and wrong-room interactions are detectable, and action efficiency is measurable.\n\n-\n02\n### Explicit social feedback\n\n4 NPCs carry trust and suspicion state (0–100) that moves in response to dialogue: feedback a model can adapt to within a run, or fail to.\n\n-\n03\n### Within-run persistence\n\nItems taken stay taken; trust earned stays earned. Every run leaves a complete, replayable transcript of exploration and planning.\n\n## The central finding is about measurement, not rankings\n\nA single LLM-judge component inside the scoring stack reordered the leaderboard by up to six positions, while every aggregate reliability statistic stayed silent. We report every result under two scoring configurations and treat the divergence as the paper's most generalizable finding.\n\n### Judge ablation reorders the top of the board\n\nTwo of four scored dimensions route through a dialogue classifier whose per-model agreement with an\nindependent judge spans **21.7% to 84.8%**, instability the aggregate κ = 0.04\nnever reveals. Removing the classifier-dependent dimensions shifts six rankings beyond scenario-sampling\nnoise (90% paired block bootstrap).\n\nThe largest mover shares a model family with the classifier. Benchmarks that use LLM judges should report per-subject agreement and ranking stability under judge ablation, not aggregate reliability alone.\n\n| Model | Full | CM | Δ |\n|---|---|---|---|\n| Claude Sonnet 4.6 | #4 | #1 | ▲ 3 |\n| DeepSeek R1 | #7 | #2 | ▲ 5 |\n| Grok 4 | #12 | #10 | ▲ 2 |\n| GPT-5.4 | #1 | #5 | ▼ 4 |\n| Gemini 3.1 Pro | #3 | #9 | ▼ 6 |\n| Mistral Large 3 | #10 | #12 | ▼ 2 |\n\n| Model | Classifier-min. | Full score | Success | $ / run |\n|---|---|---|---|---|\n| Claude Sonnet 4.6 | 4.04 | 3.89 | 24% | $0.125 |\n| DeepSeek R1 | 4.00 | 3.85 | 22% | $0.119 |\n| Claude Opus 4.6 | 3.93 | 3.93 | 30% | $0.205 |\n| GPT-5.2 | 3.91 | 3.88 | 38% | $0.113 |\n| GPT-5.4 | 3.88 | 4.07 | 68% | $0.060 |\n| Qwen 3.5 397B | 3.81 | 3.81 | 30% | $0.017 |\n| Claude Haiku 4.5 | 3.80 | 3.88 | 34% | $0.039 |\n| GPT-5.3 Chat | 3.73 | 3.72 | 40% | $0.095 |\n| Gemini 3.1 Pro | 3.71 | 3.91 | 48% | $0.339 |\n| Grok 4 | 3.61 | 3.48 | 32% | $0.834 |\n| DeepSeek V3.2 | 3.60 | 3.61 | 24% | $0.008 |\n| Mistral Large 3 | 3.44 | 3.69 | 40% | $0.017 |\n| OLMo 3.1 32B | 2.01 | 1.93 | 4% | $0.005 |\n\nMean scores on a 1–5 rubric scale, sorted by classifier-minimized subtotal. 50 runs per\nmodel: 5 seeds × 2 objectives × 5 repetitions, temperature 0.3, billing-verified via OpenRouter. Rankings\nare exploratory; confidence intervals overlap substantially among the top eight. Full protocol, CIs, and\nstatistics in the [whitepaper](whitepaper.pdf).\n\n## Failures you can read in the transcript\n\nThree failure modes, each detected algorithmically from state-machine telemetry, with no judge involved. Dialogue looping is the dominant mode for every model tested, frontier included.\n\n-\n### Dialogue looping 14–66% of frontier runs\n\nEight or more talk commands at a single NPC in one run. The agent repeats a failed conversational approach instead of adapting: the persistent-world cousin of a support agent repeating itself.\n\n-\n### Wrong-room interaction severe in floor model\n\nA talk command answered by \"no one here.\" Reveals lost world-state tracking, analogous to calling an API that is not in scope. Grok 4 was the only frontier model with meaningful incidence (12%).\n\n-\n### Exploration paralysis selective, floor-dominant\n\nTwo or fewer rooms across twenty-plus turns, or five consecutive look commands. Information gathering that never becomes goal-directed action.\n\n## What this is and is not\n\n### This is\n\n- A proof-of-concept for persistent-world behavioral evaluation.\n- A compact MUD with hidden social objectives and rule-based mechanics.\n- A way to surface measurable, interpretable failure modes.\n- A full artifact release: 650 transcripts, source, scoring code, and the complete billing export.\n\n### This is not\n\n- A validated measure of general social intelligence.\n- A definitive leaderboard of frontier models.\n- Yet predictive of real-world agent deployment outcomes.\n- A claim that LLM judges are useless (rather, evidence they need per-subject audits).\n\n## Phase 2 is where this becomes a benchmark. Help us build it.\n\nCrucibleBench is an independent research effort. Phase 2 is being built for calibration; a provisional low/base/high budget is published now, and the final allocation will follow pilot data and preregistration. There are three ways in.\n\nfund it · provisional $3,500 envelope build it · environment, objectives, calibration run it · post-calibration pilot cohort\n\nQuestions, or interested in a private evaluation? Write to\n[contact@cruciblebench.ai](mailto:contact@cruciblebench.ai)", "url": "https://wpnews.pro/news/can-a-mud-evaluate-llms-a-99-proof-of-concept", "canonical_source": "https://cruciblebench.ai/", "published_at": "2026-07-22 15:39:01+00:00", "updated_at": "2026-07-22 15:52:37.000238+00:00", "lang": "en", "topics": ["large-language-models", "ai-research"], "entities": ["CrucibleBench", "Claude Sonnet 4.6", "DeepSeek R1", "Grok 4", "GPT-5.4", "Gemini 3.1 Pro", "Mistral Large 3", "Claude Opus 4.6"], "alternates": {"html": "https://wpnews.pro/news/can-a-mud-evaluate-llms-a-99-proof-of-concept", "markdown": "https://wpnews.pro/news/can-a-mud-evaluate-llms-a-99-proof-of-concept.md", "text": "https://wpnews.pro/news/can-a-mud-evaluate-llms-a-99-proof-of-concept.txt", "jsonld": "https://wpnews.pro/news/can-a-mud-evaluate-llms-a-99-proof-of-concept.jsonld"}}