UncertaintyGym: A Benchmark for LLM Epistemic Calibration and Uncertainty Expression

Muse Ltd. released UncertaintyGym, a benchmark for evaluating large language models' epistemic calibration and uncertainty expression, available on Hugging Face under the Apache-2.0 license. The benchmark measures whether models explicitly express uncertainty, request missing context, or reject false premises across four categories: solvable, under-specified, false premise, and inherently unknowable. Initial baseline results show LiquidAI's LFM2.5-2.6B model achieved a Meta Cognitive Calibration Score of 45.0%, with a hallucination rate of 73.3%.

Hi Hugging Face team, We would like to submit UncertaintyGym for official benchmark indexing and Evaluation Hub integration. Overview: UncertaintyGym evaluates an LLM’s meta-cognitive calibration—measuring whether models explicitly express uncertainty “I don’t know” , request missing context, or reject false premises instead of hallucinating. Taxonomy: - Category A Solvable : Standard factual queries control set . - Category B Under-specified : Queries lacking necessary context that require disambiguation. - Category C False Premise : Queries with impossible premises that require rejection. - Category D Inherently Unknowable : Unsolved conjectures, lost history, and future events requiring explicit declaration of unknowability. Technical Specifications: - Repository: Muse-Ltd/UncertaintyGym · Datasets at Hugging Face https://huggingface.co/datasets/Muse-Ltd/UncertaintyGym - Configuration: eval.yaml and lighteval task.py included for direct LightEval / LM-Evaluation-Harness integration. - Primary Metric: Meta Cognitive Calibration Score MCS and Unanswerable Hallucination Rate. - License: Apache-2.0 Initial Baseline: - LiquidAI / LFM2.5-2.6B: 45.0% MCS Cat A: 100%, Cat B: 20%, Cat C: 0%, Cat D: 60%, Hallucination Rate: 73.3% Please let us know if any additional metadata is required for the benchmark badge. Thank you.