The Two MMLU Scores: What a Benchmark Name Does Not Fix Two MMLU accuracy scores of 0.781 for build 42 and 0.79 for build 44 of the acme-gpt-7b model family are structurally valid but return "incomparable" from the score-delta verifier in the apl-ai-eval crate because their frames declare different runners, graders and dataset splits. Author Dmitrii Zatona writes that the shared mmlu label "identifies a dataset family, not a full measurement procedure," and that comparability attaches to the reference results are traceable to rather than to the number itself. Under the APL AI-Eval profile the frame is a content-addressed object carrying its hash, and the two frames differ, with subset: "all" and an omitted key treated as different scopes by canonical bytes. The Two MMLU Scores: What a Benchmark Name Does Not Fix Date: September 6, 2026 · Author: Dmitrii Zatona https://zatona.dev/ TL;DR - Two MMLU accuracies, 0.781 and 0.79 , for two builds of one model family under the same benchmark name; for a score-delta query the verifier returns incomparable Sections 1 and 5 . - mmlu fixes a name. The split, the implementation, the prompt format, the grader and the runner’s network access stay open, and where published measurements exist for them the differences are points of accuracy, not thousandths Section 2 . - Comparability is a property of the reference the results are traceable to, not of the number Section 3 . - Under the APL AI-Eval profile the frame is a content-addressed object and the claim carries its hash; the two frames differ, and subset: "all" and an omitted key are different scopes by canonical bytes Section 4 . - apl-valid is a statement about structure and says nothing about whether either score is correct Section 6 . Two evaluation records appear in the same table. One reports mmlu accuracy 0.781 for build 42; the other reports 0.79 for build 44. The claims declare the same provider, model family, metric identifier, unit and benchmark name. The arithmetic difference is +0.009 . The records are structurally valid. Their frames declare different runners, graders and dataset splits. The shared mmlu label identifies a dataset family, not a full measurement procedure. For the score-delta query shown below, the verifier returns incomparable . What follows is that pair taken apart: what stays open once mmlu is fixed and what published numbers say each open variable is worth; why comparability attaches to the reference rather than to the number; and what the verifier in the apl-ai-eval crate outputs once the conditions are a hashed object, with a bridge and without one. Every frame and claim below is copied from the crate’s test vectors; the verifier outputs were recorded by running the crate’s public verification functions against those vectors. 1. What the two numbers say Here is claim A as it exists on the wire, in the metadata.apl position of a log entry: {"apl":{"version":"0.1","claim":{"kind":"observation","subject":{"type":"model-build","id":"model:acme-gpt-7b-build-42","build id":"42","artifact digest":"sha256:4242424242424242424242424242424242424242424242424242424242424242","provider":"acme","model family":"acme-gpt-7b"},"aspect refs": "accuracy" ,"statement":{"predicate":"score","content":{"benchmark id":"mmlu","metric id":"accuracy","value":0.781,"unit":"fraction"}}},"frame ref":{"hash":"sha256:c7b88426f2676f3653db0fad0bdbd689318f16d589d14a315bdd4cc454bca1ab"}}} Claim B, two builds later: {"apl":{"version":"0.1","claim":{"kind":"observation","subject":{"type":"model-build","id":"model:acme-gpt-7b-build-44","build id":"44","artifact digest":"sha256:4444444444444444444444444444444444444444444444444444444444444444","provider":"acme","model family":"acme-gpt-7b"},"aspect refs": "accuracy" ,"statement":{"predicate":"score","content":{"benchmark id":"mmlu","metric id":"accuracy","value":0.79,"unit":"fraction"}}},"frame ref":{"hash":"sha256:c93a9c55422ddbd2158a5336caa3a251641cf3937f451fb13387a5f54f0d998e"}}} The subject differs, which is the point: two builds of one family. Each claim declares an artifact digest , which the profile treats as the immutable identity anchor of the evaluated artifact. The statement is identical in shape and vocabulary — predicate score , benchmark mmlu , metric accuracy , unit fraction . Claim B writes the value as 0.79 ; a presentation may display that JSON value as 0.790 , and under RFC 8785 canonical number serialization the trailing zero does not change the value. The claim-level pointers differ: frame ref.hash is c7b88426… in one record and c93a9c55… in the other. Resolving the two frames shows differences in both procedure and scope; the hash is the whole signal at the claim level. The relation someone wants over this pair is also an object: {"left aspects": "accuracy" ,"right aspects": "accuracy" ,"predicate":"score","relation type":"score-delta"} score-delta is a request to subtract. What has to hold for that request to have an answer is Sections 2 and 3; what the verifier returns when it does not is Section 5. 2. What “MMLU” does not fix benchmark id: mmlu fixes a name. Five things it leaves open, and what the record says each is worth. 2.1 The split The MMLU paper reports 15,908 questions split into a few-shot development set of 5 questions per subject across 57 subjects, a validation set of 1,540 and a test set of 14,079 Hendrycks et al., arXiv:2009.03300 https://arxiv.org/abs/2009.03300 , §3 . The Hugging Face dataset most runners load, cais/mmlu config all , reports test 14,042, validation 1,531, dev 285 cais/mmlu https://huggingface.co/datasets/cais/mmlu . The archive linked from the hendrycks/test https://github.com/hendrycks/test README was not retrievable when checked in September 2026 HTTP 403 after redirect . “The MMLU test set” names two objects of different sizes, and no reviewed document explains the difference. Frame A scores on dev , the 285 questions the paper defines as the source of its fixed few-shot examples §4.1 , so scoring on it is a choice, and the kind of choice a frame should make visible. Frame B scores on test-lite , the label used in the crate’s own test vectors for a reduced set; a check of the Hugging Face datasets and models APIs and of GitHub repository search found no published artifact under that name, and it is not tinyMMLU https://arxiv.org/abs/2402.14992 . The label tells a reader a private slice was used. It does not tell them which questions. 2.2 The implementation One published measurement of implementation variance is the June 2023 Hugging Face post on the Open LLM Leaderboard. Three harnesses — HELM, the Eleuther harness, the original code — run the same dataset, all 5-shot, and score llama-65b at 0.637, 0.488 and 0.636; falcon-40b at 0.571, 0.527 and 0.558. The post concludes that the three results are not comparable despite the shared MMLU label What’s going on with the Open LLM Leaderboard? https://huggingface.co/blog/open-llm-leaderboard-mmlu . The mechanism is scoring. The original code compares the probabilities of the four answer letters; HELM generates from the next-token output and compares to expected text; the harness scores the full answer sequence including the option text. Rank order moves with it — falcon-40b sits above llama-65b under the harness and below it under the other two. The runner is not one object either. The lm-evaluation-harness MMLU README describes mmlu , mmlu continuation cloze-style and mmlu generative the model produces the answer letter as three tasks over the same data, at different task versions lm-evaluation-harness https://github.com/EleutherAI/lm-evaluation-harness , commit b954108c . Its task guide treats the YAML config plus the codebase commit hash as the unit another researcher needs to replicate a setup; num fewshot defaults to 0, and the MMLU YAML sets none — so “5-shot MMLU” is a command-line flag, not a property of the task. Version 0.3.0 asked users to report each task’s version; current main carries no such request. 2.3 The prompt format Anthropic’s 2023 account of evaluation reports that formatting alone — option labels, parentheses, an extra space before the answer — moves MMLU accuracy by about 5% Challenges in evaluating AI systems https://www.anthropic.com/news/evaluating-ai-systems . Answer position moves more. Zheng et al. report that on MMLU, moving the correct answers to position D lowers gpt-3.5-turbo from 67.2 to 60.9, and that moving them to A lifts llama-30b by 15.2 points to 68.2 against gpt-3.5-turbo’s 65.3, reversing the original 53.1 against 67.2 arXiv:2309.03882 https://arxiv.org/abs/2309.03882 . Both frames here declare prompt protocol: zero-shot-mcq-v1 , so this variable is held. A held variable is only visibly held if it is written down. 2.4 The grader Frame A grades with exact-match-v1 . Frame B grades with llm-judge-v3 . Those are not two implementations of one function. Zheng et al. found that judge models “exhibit strong position bias”, that only GPT-4 stayed consistent in more than 60% of cases — 65.0%, against 46.2% for GPT-3.5 and 23.8% for Claude-v1 — and describe a judgement that flips when two responses swap positions arXiv:2306.05685 https://arxiv.org/abs/2306.05685 , Table 2 . The same paper puts GPT-4–human agreement at 85% against 81% human–human. It is a different grading procedure with a different documented failure mode. 2.5 What the runner could reach Another variable is the environment the runner was allowed to touch. CAISI published an account of finding, after the fact, that it had been running SWE-bench Verified with internet access while other evaluators ran without it, and that it learned this from transcripts other evaluators had posted rather than from the benchmark’s documentation Cheating on AI Agent Evaluations https://www.nist.gov/caisi/cheating-ai-agent-evaluations , December 2025 . Its new policy for coding evaluations is “fully offline”. Reachability matters for static benchmarks too: Scale’s search-time contamination work found roughly 3% of questions retrievable with labels from Hugging Face, and blocking that source cut accuracy on the contaminated subset by about 15 points Search-Time Data Contamination https://labs.scale.com/papers/stc . Where CAISI compares its own results to self-reported ones — SWE-bench Verified at 63.0 against 74.9 self-reported for one model — it lists possible sources of the differences, including dataset differences, agent setup and API sampling parameters such as temperature and top p CAISI Evaluation of DeepSeek AI Models https://www.nist.gov/system/files/documents/2025/09/30/CAISI Evaluation of DeepSeek AI Models.pdf , Appendix A8 . | Variable | What the record says | Effect on the number | |---|---|---| | Split | Paper: test 14079; cais/mmlu all : test 14042, dev 285 | Different denominators under one name; dev is a few-shot source by design | | Implementation | HF blog: HELM / harness / original, all 5-shot | llama-65b 0.637 / 0.488 / 0.636; rank order flips | | Task variant, shot count | Harness README and task guide: three MMLU tasks; num fewshot defaults to 0, MMLU YAML sets none | Different scoring targets under one dataset; “5-shot” lives in the invocation | | Prompt format | Anthropic 2023; Sclar et al., arXiv:2310.11324; Alzahrani et al., arXiv:2402.01781 | ~5% from punctuation; up to 76 accuracy points from format; ranking shifts up to 8 positions | | Answer position | Zheng et al., arXiv:2309.03882 | gpt-3.5-turbo 67.2 → 60.9; llama-30b +15.2 | | Grader | Zheng et al., arXiv:2306.05685 | Judge self-consistency 65.0% / 46.2% / 23.8% across three judges | | Network access | CAISI, SWE-bench Verified; Scale STC | Found post hoc from other evaluators’ transcripts; ~15 points on a contaminated subset | None of this is exotic. A 2026 survey of 101,843 published results across 5,816 models and 635 benchmarks found that 48,698 96.5% lack at least one field from its minimal reproducibility sub-schema, that temperature is absent from 93.9%, and recorded one model reported at a 20.9% score by Hugging Face and 61.8% by a second evaluator on MMLU-Pro Evaluation Cards, arXiv:2606.09809 https://arxiv.org/abs/2606.09809 . The survey documents incomplete reporting at scale; it does not say how often this exact combination of differences occurs. 3. Comparability is a property of the reference, not of the number VIM §2.46 defines metrological comparability of measurement results through traceability to a common reference, as comparability of results, for quantities of a given kind, “that are metrologically traceable to the same reference” JCGM 200:2012 . Its §2.41 note permits a measurement procedure to serve as that reference for a non-ordinal quantity. The AI-Eval profile turns that into a narrower operational rule: a bridge is applicable only when the frames meet its exact aspect, scope and procedure constraints, and mmlu alone does not meet them. Metrological compatibility, §2.47, is a separate VIM term: it concerns whether the difference between two results is small relative to the uncertainty of that difference. It does not establish that two results share a reference. Fields outside AI have already paid for this. HbA1c has two reference methods, and NGSP reports IFCC results running 1.5 to 2% HbA1c lower across the range NGSP http://www.ngsp.org/ifccngsp.asp ; in 2007 the IFCC value was given its own unit, mmol/mol, which NGSP says avoids confusion between the two systems. Cloud availability has the same shape: AWS EC2 derives Monthly Uptime Percentage from minutes of Unavailability, defined as loss of external connectivity AWS EC2 SLA https://aws.amazon.com/compute/sla/ , while Google Compute counts loss of persistent disk access as Downtime and does not count intermittent downtime shorter than one minute Google Compute Engine SLA https://cloud.google.com/compute/sla . Two contracts print “99.99%” over different measurands. The AI-specific documents arrive at the same statement without the metrology words. NIST’s draft practices for automated benchmark evaluations tie comparability to consistency of protocol between systems §2.1 and describe breaking changes under semantic versioning as points after which results on either side are no longer properly comparable NIST AI 800-2 ipd https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.800-2.ipd.pdf , January 2026 . Practice 2.3 lists grouping together the evaluation logs that are meant to be compared as an emerging practice, and Practice 3.2 notes that an interoperable schema for sharing evaluation details may ease replication. The NAAIMES network best practice of July 2026 splits the property in two, the evaluation of outputs scoring criteria, rubrics, graders and the generation of outputs prompt format, inference settings , holds that the first should always be comparable, and treats the second as depending on the goals of the evaluation §C.1.1 . grader id records the grading procedure, while prompt protocol records part of output generation; the two frames differ in both of those fields and in runner id . An earlier post here /blog/ai-agent-audit-trail put the general form in one line — a trail that records “accuracy 0.78” without the frame records a number, not an observation. The two MMLU scores are that line’s specific case with the hashes filled in. 4. Binding the number to its frame Under APL /apl , the frame is not documentation attached to a claim but a separate content-addressed object, and the claim carries only its hash. The AI-Eval profile — the vocabulary for benchmark observations about a model build — fixes what that object must contain. Frame A, indented for reading: { "version": "0.1", "observer": { "id": "acme-eval-lab" }, "procedure": { "runner id": " email protected ", "grader id": "exact-match-v1", "prompt protocol": "zero-shot-mcq-v1" }, "aspect": "accuracy" , "scope": { "benchmark id": "mmlu", "benchmark variant": "default", "dataset split": "dev", "subset": "all" }, "invariance": "score-object-serialization" , "exclusions": "no-production-readiness-claim", "no-deployment-safety-claim", "no-out-of-scope-generalization-claim" } Its hash is sha256:c7b88426f2676f3653db0fad0bdbd689318f16d589d14a315bdd4cc454bca1ab , and the indentation above is display only: the hash is SHA-256 over the RFC 8785 canonical bytes src/core/jcs.rs:167 , canonical hash , so whitespace never reaches the digest. Frame B is the same object with runner id at \ email protected\ /cdn-cgi/l/email-protection grader id at llm-judge-v3 , dataset split at test-lite and no subset key, hashing to sha256:c93a9c55422ddbd2158a5336caa3a251641cf3937f451fb13387a5f54f0d998e . The profile validator turns those fields from convention into a requirement. procedure must be an object with a non-empty runner id frame.rs:35 and grader id frame.rs:36 ; scope must carry benchmark id , benchmark variant and dataset split frame.rs:48–50 ; aspect must hold exactly one entry from a closed list frame.rs:51–57 ; exclusions must contain all three markers, which is how a frame states what it is not evidence of frame.rs:69–76 . On the claim side, content.benchmark id is required claim.rs:265 and must equal frame.scope.benchmark id claim.rs:106–121 , and content.metric id is required and constrained by the aspect — accuracy admits accuracy , exact-match or f1 claim.rs:268 , 315–318 . The distance between a description and a validator is visible in the project’s own history. The design document that introduced this pair writes procedure as a human-readable string, invariance as “score object serialization” with spaces, the exclusions as two sentences of prose, the claim with content.benchmark and no metric id , and the hashes as placeholders. The document’s examples use fields the current crate rejects. A prose value has no controlled vocabulary; different strings serialize differently and cannot satisfy exact equality. A subtler case sits inside the two frames. Frame A writes subset: "all" ; frame B omits the key. A reader might expect the comparison to normalize that, an absent optional field standing for its default. It does not. Scope equality is decided by canonical bytes bridge.rs:114–118 calling canonical equal , implemented at src/core/jcs.rs:84–87 as canonical bytes a == canonical bytes b , and serde’s derived PartialEq is explicitly not used for it. Under that rule “all” and absent are two scopes. The rule leaves defaulting to the tool that emits the frame rather than to the comparison step. 5. What the verifier says apl-ai-eval is a Rust crate, version 0.3.1, Apache-2.0. Given a receipt or a pair of receipts and a relation query, it decides whether the claims are well-formed under the profile and whether the relation is evaluable. The case in this post lives in its test vectors as two mmlu scores incomparable with ai eval profile , with the pair in test data/vectors/pairwise/incomparable-no-bridge.json and two sibling vector files containing bridge cases. cargo test -p apl-ai-eval runs 109 unit tests, 3 vector integration tests and 2 doc-tests; all 114 are green. Verify claim A on its own against the AI-Eval profile and the output is: {"core outcome":"apl-valid","relation outcome":"relation-not-evaluated","failure classes": ,"diagnostics": "carrier-valid","apl-present","apl-frame-bound","apl-valid","same-frame","transformation-missing" } Claim B alone produces the same shape. apl-valid here means well-formed and frame-bound, and says nothing about the accuracy of 0.781 ; relation-not-evaluated is the answer to a question that was not asked. Now the pair, with the score-delta query from Section 1 and no bridge: {"left":{"core outcome":"apl-valid","failure classes": },"right":{"core outcome":"apl-valid","failure classes": },"relation outcome":"incomparable","diagnostics": "apl-cross-frame","apl-bridge-not-found" } Both claims have core outcome: "apl-valid" and empty failure classes; the relation outcome is incomparable . apl-cross-frame states that the two frame ref hashes differ; apl-bridge-not-found , that nothing was supplied to license a comparison across that difference. The two axes are separate by design: this is not a broken record and not a passing comparison. It is a subtraction with no defined result, reported as such. The third vector is a separate adversarial case, not a bridge between Frames A and B above. Its source frame declares aspect accuracy and its target frame declares aspect judge-score . The bridge is itself a content-addressed object, and this one is well-formed: {"version":"0.1","bridge kind":"runner-equivalence","source frame":{"hash":"