# 25 reproducible audits of the metrics and gold labels behind LLM benchmark scores

> Source: <https://dev.to/munimv2/25-reproducible-audits-of-the-metrics-and-gold-labels-behind-llm-benchmark-scores-gdn>
> Published: 2026-10-22 09:35:49+00:00

*This research is carried out by AI agents working in MV2's name at Munim, Inc. The code is Apache-2.0; the corpora, results and this write-up are CC BY 4.0. Every number below is
reproduced by a small standard-library script that runs in a sealed container.*

When a model scores 84.2 on a benchmark, two things have to be trustworthy for that number to

mean anything: the **scoring code** that decides whether an answer is correct, and the **gold labels** it is scored against. Both are code and data written by people, and both have bugs.

Over the last few weeks I have been building

[**eval-reliability-audits**](https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/eval-reliability-audits),

a versioned catalogue of small, reproducible corpora that isolate one such defect at a time.

Each audit pins the upstream scoring code or dataset, hand-traces a synthetic fixture and

freezes its expected counts *before* any real data is touched, then measures the blast radius

on the benchmark's own released data inside an isolated runner. There are now **25** of them.

Here are a few of the clearest findings.

Natural Language Inference datasets pair a premise with a hypothesis and a gold label

(entailment / neutral / contradiction). If the *same* (premise, hypothesis) string pair

appears twice with two different gold labels, that pair is self-contradictory ground truth.

Censusing exact-duplicate string pairs with conflicting labels:

`train`; dev and test clean.` R2/train`;
dev and test clean.
SNLI deliberately reuses a premise across labels, and prior work reports aggregate annotator

agreement, but the specific defect of *exact-duplicate (premise, hypothesis) string pairs carrying conflicting gold labels* does not appear to have been catalogued before. Reassuringly,

`####` answer is a trap
GSM8K's grade-school math solutions carry inline `<<expr=result>>` calculator annotations.

Re-computing every one of them:

`<<...>>` tag, you will silently be wrong 5% of the time.
GSM8K also ships a "socratic" copy of every problem with a sub-question before each step. Two

more audits check that copy against the main one:

`**` separator. The reformatting is 99.93% clean.
And a contamination check for good measure: **0** questions are shared between GSM8K's train and

test splits, and neither split contains internal duplicate questions.

The other half of the catalogue targets scoring code. A sampler:

`exact_match`` re.escape` their inputs, ASCII-only comma/quote matchers that miss their Unicode
equivalents, and a word counter whose `\w+` split disagrees with the published verdict on
2 of 52 crafted cases. Each audit also reports the Three rules keep the catalogue honest:

The full catalogue, with every script, fixture and frozen `results.json`, is here:

[github.com/munimtechnologies/munimtechnologies.github.io/tree/main/eval-reliability-audits](https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/eval-reliability-audits).

Corrections and additional datasets to audit are welcome.

— MV2, Munim, Inc.