# UncertaintyGym: A Benchmark for LLM Epistemic Calibration and Uncertainty Expression

> Source: <https://discuss.huggingface.co/t/uncertaintygym-a-benchmark-for-llm-epistemic-calibration-and-uncertainty-expression/178647#post_1>
> Published: 2026-08-13 19:26:31+00:00

Hi Hugging Face team,

We would like to submit UncertaintyGym for official benchmark indexing and Evaluation Hub integration.

Overview:

UncertaintyGym evaluates an LLM’s meta-cognitive calibration—measuring whether models explicitly express uncertainty (“I don’t know”), request missing context, or reject false premises instead of hallucinating.

Taxonomy:

- Category A (Solvable): Standard factual queries (control set).

- Category B (Under-specified): Queries lacking necessary context that require disambiguation.

- Category C (False Premise): Queries with impossible premises that require rejection.

- Category D (Inherently Unknowable): Unsolved conjectures, lost history, and future events requiring explicit declaration of unknowability.

Technical Specifications:

- Repository: [Muse-Ltd/UncertaintyGym · Datasets at Hugging Face](https://huggingface.co/datasets/Muse-Ltd/UncertaintyGym)

- Configuration: eval.yaml and lighteval_task.py included for direct LightEval / LM-Evaluation-Harness integration.

- Primary Metric: Meta Cognitive Calibration Score (MCS) and Unanswerable Hallucination Rate.

- License: Apache-2.0

Initial Baseline:

- LiquidAI / LFM2.5-2.6B: 45.0% MCS (Cat A: 100%, Cat B: 20%, Cat C: 0%, Cat D: 60%, Hallucination Rate: 73.3%)

Please let us know if any additional metadata is required for the benchmark badge.

Thank you.