UncertaintyGym: A Benchmark for LLM Epistemic Calibration and Uncertainty Expression

Hi Hugging Face team,

We would like to submit UncertaintyGym for official benchmark indexing and Evaluation Hub integration.

Overview:

UncertaintyGym evaluates an LLM’s meta-cognitive calibration—measuring whether models explicitly express uncertainty (“I don’t know”), request missing context, or reject false premises instead of hallucinating.

Taxonomy:

- Category A (Solvable): Standard factual queries (control set).

- Category B (Under-specified): Queries lacking necessary context that require disambiguation.

• Category C (False Premise): Queries with impossible premises that require rejection.

• Category D (Inherently Unknowable): Unsolved conjectures, lost history, and future events requiring explicit declaration of unknowability.

Technical Specifications:

• Repository: Muse-Ltd/UncertaintyGym · Datasets at Hugging Face

• Configuration: eval.yaml and lighteval_task.py included for direct LightEval / LM-Evaluation-Harness integration.

• Primary Metric: Meta Cognitive Calibration Score (MCS) and Unanswerable Hallucination Rate.

• License: Apache-2.0 Initial Baseline:

• LiquidAI / LFM2.5-2.6B: 45.0% MCS (Cat A: 100%, Cat B: 20%, Cat C: 0%, Cat D: 60%, Hallucination Rate: 73.3%) Please let us know if any additional metadata is required for the benchmark badge.

Thank you.