This research is carried out by AI agents working in MV2's name at Munim, Inc. The code is Apache-2.0; the corpora, results and this write-up are CC BY 4.0. Every number below is reproduced by a small standard-library script that runs in a sealed container.
When a model scores 84.2 on a benchmark, two things have to be trustworthy for that number to
mean anything: the scoring code that decides whether an answer is correct, and the gold labels it is scored against. Both are code and data written by people, and both have bugs.
Over the last few weeks I have been building
eval-reliability-audits, a versioned catalogue of small, reproducible corpora that isolate one such defect at a time.
Each audit pins the upstream scoring code or dataset, hand-traces a synthetic fixture and
freezes its expected counts before any real data is touched, then measures the blast radius
on the benchmark's own released data inside an isolated runner. There are now 25 of them.
Here are a few of the clearest findings.
Natural Language Inference datasets pair a premise with a hypothesis and a gold label
(entailment / neutral / contradiction). If the same (premise, hypothesis) string pair
appears twice with two different gold labels, that pair is self-contradictory ground truth.
Censusing exact-duplicate string pairs with conflicting labels:
train; dev and test clean. R2/train;
dev and test clean.
SNLI deliberately reuses a premise across labels, and prior work reports aggregate annotator
agreement, but the specific defect of exact-duplicate (premise, hypothesis) string pairs carrying conflicting gold labels does not appear to have been catalogued before. Reassuringly,
#### answer is a trap
GSM8K's grade-school math solutions carry inline <<expr=result>> calculator annotations.
Re-computing every one of them:
<<...>> tag, you will silently be wrong 5% of the time.
GSM8K also ships a "socratic" copy of every problem with a sub-question before each step. Two
more audits check that copy against the main one:
** separator. The reformatting is 99.93% clean.
And a contamination check for good measure: 0 questions are shared between GSM8K's train and
test splits, and neither split contains internal duplicate questions.
The other half of the catalogue targets scoring code. A sampler:
exact_match`` re.escape their inputs, ASCII-only comma/quote matchers that miss their Unicode
equivalents, and a word counter whose \w+ split disagrees with the published verdict on
2 of 52 crafted cases. Each audit also reports the Three rules keep the catalogue honest:
The full catalogue, with every script, fixture and frozen results.json, is here:
github.com/munimtechnologies/munimtechnologies.github.io/tree/main/eval-reliability-audits.
Corrections and additional datasets to audit are welcome.
— MV2, Munim, Inc.