{"slug": "the-two-mmlu-scores-what-a-benchmark-name-does-not-fix", "title": "The Two MMLU Scores: What a Benchmark Name Does Not Fix", "summary": "Two MMLU accuracy scores of 0.781 for build 42 and 0.79 for build 44 of the acme-gpt-7b model family are structurally valid but return \"incomparable\" from the score-delta verifier in the apl-ai-eval crate because their frames declare different runners, graders and dataset splits. Author Dmitrii Zatona writes that the shared mmlu label \"identifies a dataset family, not a full measurement procedure,\" and that comparability attaches to the reference results are traceable to rather than to the number itself. Under the APL AI-Eval profile the frame is a content-addressed object carrying its hash, and the two frames differ, with subset: \"all\" and an omitted key treated as different scopes by canonical bytes.", "body_md": "# The Two MMLU Scores: What a Benchmark Name Does Not Fix\n\n**Date:** September 6, 2026 · **Author:** [Dmitrii Zatona](https://zatona.dev/) \n\n## TL;DR\n\n- Two MMLU accuracies, `0.781` and`0.79` , for two builds of one model family under the same benchmark name; for a`score-delta` query the verifier returns`incomparable` (Sections 1 and 5).\n- `mmlu` fixes a name. The split, the implementation, the prompt format, the grader and the runner’s network access stay open, and where published measurements exist for them the differences are points of accuracy, not thousandths (Section 2).\n- Comparability is a property of the reference the results are traceable to, not of the number (Section 3).\n- Under the APL AI-Eval profile the frame is a content-addressed object and the claim carries its hash; the two frames differ, and `subset: \"all\"` and an omitted key are different scopes by canonical bytes (Section 4).\n- `apl-valid` is a statement about structure and says nothing about whether either score is correct (Section 6).\n\nTwo evaluation records appear in the same table. One reports `mmlu` accuracy `0.781` for build 42; the other reports `0.79` for build 44. The claims declare the same provider, model family, metric identifier, unit and benchmark name. The arithmetic difference is `+0.009`.\n\nThe records are structurally valid. Their frames declare different runners, graders and dataset splits. The shared `mmlu` label identifies a dataset family, not a full measurement procedure. For the `score-delta` query shown below, the verifier returns `incomparable`.\n\nWhat follows is that pair taken apart: what stays open once `mmlu` is fixed and what published numbers say each open variable is worth; why comparability attaches to the reference rather than to the number; and what the verifier in the `apl-ai-eval` crate outputs once the conditions are a hashed object, with a bridge and without one. Every frame and claim below is copied from the crate’s test vectors; the verifier outputs were recorded by running the crate’s public verification functions against those vectors.\n\n## 1. What the two numbers say\n\nHere is claim A as it exists on the wire, in the `metadata.apl` position of a log entry:\n\n```\n{\"apl\":{\"version\":\"0.1\",\"claim\":{\"kind\":\"observation\",\"subject\":{\"type\":\"model-build\",\"id\":\"model:acme-gpt-7b-build-42\",\"build_id\":\"42\",\"artifact_digest\":\"sha256:4242424242424242424242424242424242424242424242424242424242424242\",\"provider\":\"acme\",\"model_family\":\"acme-gpt-7b\"},\"aspect_refs\":[\"accuracy\"],\"statement\":{\"predicate\":\"score\",\"content\":{\"benchmark_id\":\"mmlu\",\"metric_id\":\"accuracy\",\"value\":0.781,\"unit\":\"fraction\"}}},\"frame_ref\":{\"hash\":\"sha256:c7b88426f2676f3653db0fad0bdbd689318f16d589d14a315bdd4cc454bca1ab\"}}}\n```\n\nClaim B, two builds later:\n\n```\n{\"apl\":{\"version\":\"0.1\",\"claim\":{\"kind\":\"observation\",\"subject\":{\"type\":\"model-build\",\"id\":\"model:acme-gpt-7b-build-44\",\"build_id\":\"44\",\"artifact_digest\":\"sha256:4444444444444444444444444444444444444444444444444444444444444444\",\"provider\":\"acme\",\"model_family\":\"acme-gpt-7b\"},\"aspect_refs\":[\"accuracy\"],\"statement\":{\"predicate\":\"score\",\"content\":{\"benchmark_id\":\"mmlu\",\"metric_id\":\"accuracy\",\"value\":0.79,\"unit\":\"fraction\"}}},\"frame_ref\":{\"hash\":\"sha256:c93a9c55422ddbd2158a5336caa3a251641cf3937f451fb13387a5f54f0d998e\"}}}\n```\n\nThe subject differs, which is the point: two builds of one family. Each claim declares an `artifact_digest`, which the profile treats as the immutable identity anchor of the evaluated artifact. The statement is identical in shape and vocabulary — predicate `score`, benchmark `mmlu`, metric `accuracy`, unit `fraction`. Claim B writes the value as `0.79`; a presentation may display that JSON value as `0.790`, and under RFC 8785 canonical number serialization the trailing zero does not change the value.\n\nThe claim-level pointers differ: `frame_ref.hash` is `c7b88426…` in one record and `c93a9c55…` in the other. Resolving the two frames shows differences in both procedure and scope; the hash is the whole signal at the claim level.\n\nThe relation someone wants over this pair is also an object:\n\n```\n{\"left_aspects\":[\"accuracy\"],\"right_aspects\":[\"accuracy\"],\"predicate\":\"score\",\"relation_type\":\"score-delta\"}\n```\n\n`score-delta` is a request to subtract. What has to hold for that request to have an answer is Sections 2 and 3; what the verifier returns when it does not is Section 5.\n\n## 2. What “MMLU” does not fix\n\n`benchmark_id: mmlu` fixes a name. Five things it leaves open, and what the record says each is worth.\n\n### 2.1 The split\n\nThe MMLU paper reports 15,908 questions split into a few-shot development set of 5 questions per subject across 57 subjects, a validation set of 1,540 and a test set of 14,079 ([Hendrycks et al., arXiv:2009.03300](https://arxiv.org/abs/2009.03300) , §3). The Hugging Face dataset most runners load, `cais/mmlu` config `all`, reports test 14,042, validation 1,531, dev 285 ([cais/mmlu](https://huggingface.co/datasets/cais/mmlu) ). The archive linked from the [hendrycks/test](https://github.com/hendrycks/test)  README was not retrievable when checked in September 2026 (HTTP 403 after redirect). “The MMLU test set” names two objects of different sizes, and no reviewed document explains the difference.\n\nFrame A scores on `dev`, the 285 questions the paper defines as the source of its fixed few-shot examples (§4.1), so scoring on it is a choice, and the kind of choice a frame should make visible. Frame B scores on `test-lite`, the label used in the crate’s own test vectors for a reduced set; a check of the Hugging Face datasets and models APIs and of GitHub repository search found no published artifact under that name, and it is not [tinyMMLU](https://arxiv.org/abs/2402.14992) . The label tells a reader a private slice was used. It does not tell them which questions.\n\n### 2.2 The implementation\n\nOne published measurement of implementation variance is the June 2023 Hugging Face post on the Open LLM Leaderboard. Three harnesses — HELM, the Eleuther harness, the original code — run the same dataset, all 5-shot, and score llama-65b at 0.637, 0.488 and 0.636; falcon-40b at 0.571, 0.527 and 0.558. The post concludes that the three results are not comparable despite the shared MMLU label ([What’s going on with the Open LLM Leaderboard?](https://huggingface.co/blog/open-llm-leaderboard-mmlu) ). The mechanism is scoring. The original code compares the probabilities of the four answer letters; HELM generates from the next-token output and compares to expected text; the harness scores the full answer sequence including the option text. Rank order moves with it — falcon-40b sits above llama-65b under the harness and below it under the other two.\n\nThe runner is not one object either. The lm-evaluation-harness MMLU README describes `mmlu`, `mmlu_continuation` (cloze-style) and `mmlu_generative` (the model produces the answer letter) as three tasks over the same data, at different task versions ([lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) , commit `b954108c`). Its task guide treats the YAML config plus the codebase commit hash as the unit another researcher needs to replicate a setup; `num_fewshot` defaults to 0, and the MMLU YAML sets none — so “5-shot MMLU” is a command-line flag, not a property of the task. Version 0.3.0 asked users to report each task’s version; current main carries no such request.\n\n### 2.3 The prompt format\n\nAnthropic’s 2023 account of evaluation reports that formatting alone — option labels, parentheses, an extra space before the answer — moves MMLU accuracy by about 5% ([Challenges in evaluating AI systems](https://www.anthropic.com/news/evaluating-ai-systems) ). Answer position moves more. Zheng et al. report that on MMLU, moving the correct answers to position D lowers gpt-3.5-turbo from 67.2 to 60.9, and that moving them to A lifts llama-30b by 15.2 points to 68.2 against gpt-3.5-turbo’s 65.3, reversing the original 53.1 against 67.2 ([arXiv:2309.03882](https://arxiv.org/abs/2309.03882) ).\n\nBoth frames here declare `prompt_protocol: zero-shot-mcq-v1`, so this variable is held. A held variable is only visibly held if it is written down.\n\n### 2.4 The grader\n\nFrame A grades with `exact-match-v1`. Frame B grades with `llm-judge-v3`. Those are not two implementations of one function. Zheng et al. found that judge models “exhibit strong position bias”, that only GPT-4 stayed consistent in more than 60% of cases — 65.0%, against 46.2% for GPT-3.5 and 23.8% for Claude-v1 — and describe a judgement that flips when two responses swap positions ([arXiv:2306.05685](https://arxiv.org/abs/2306.05685) , Table 2). The same paper puts GPT-4–human agreement at 85% against 81% human–human. It is a different grading procedure with a different documented failure mode.\n\n### 2.5 What the runner could reach\n\nAnother variable is the environment the runner was allowed to touch. CAISI published an account of finding, after the fact, that it had been running SWE-bench Verified with internet access while other evaluators ran without it, and that it learned this from transcripts other evaluators had posted rather than from the benchmark’s documentation ([Cheating on AI Agent Evaluations](https://www.nist.gov/caisi/cheating-ai-agent-evaluations) , December 2025). Its new policy for coding evaluations is “fully offline”. Reachability matters for static benchmarks too: Scale’s search-time contamination work found roughly 3% of questions retrievable with labels from Hugging Face, and blocking that source cut accuracy on the contaminated subset by about 15 points ([Search-Time Data Contamination](https://labs.scale.com/papers/stc) ).\n\nWhere CAISI compares its own results to self-reported ones — SWE-bench Verified at 63.0 against 74.9 self-reported for one model — it lists possible sources of the differences, including dataset differences, agent setup and API sampling parameters such as temperature and top_p ([CAISI Evaluation of DeepSeek AI Models](https://www.nist.gov/system/files/documents/2025/09/30/CAISI_Evaluation_of_DeepSeek_AI_Models.pdf) , Appendix A8).\n\n| Variable | What the record says | Effect on the number | \n|---|---|---|\n| Split | Paper: test 14079; `cais/mmlu`` all` : test 14042, dev 285 | Different denominators under one name; `dev` is a few-shot source by design | \n| Implementation | HF blog: HELM / harness / original, all 5-shot | llama-65b 0.637 / 0.488 / 0.636; rank order flips | \n| Task variant, shot count | Harness README and task guide: three MMLU tasks; `num_fewshot` defaults to 0, MMLU YAML sets none | Different scoring targets under one dataset; “5-shot” lives in the invocation | \n| Prompt format | Anthropic 2023; Sclar et al., arXiv:2310.11324; Alzahrani et al., arXiv:2402.01781 | ~5% from punctuation; up to 76 accuracy points from format; ranking shifts up to 8 positions | \n| Answer position | Zheng et al., arXiv:2309.03882 | gpt-3.5-turbo 67.2 → 60.9; llama-30b +15.2 | \n| Grader | Zheng et al., arXiv:2306.05685 | Judge self-consistency 65.0% / 46.2% / 23.8% across three judges | \n| Network access | CAISI, SWE-bench Verified; Scale STC | Found post hoc from other evaluators’ transcripts; ~15 points on a contaminated subset | \n\nNone of this is exotic. A 2026 survey of 101,843 published results across 5,816 models and 635 benchmarks found that 48,698 (96.5%) lack at least one field from its minimal reproducibility sub-schema, that temperature is absent from 93.9%, and recorded one model reported at a 20.9% score by Hugging Face and 61.8% by a second evaluator on MMLU-Pro ([Evaluation Cards, arXiv:2606.09809](https://arxiv.org/abs/2606.09809) ). The survey documents incomplete reporting at scale; it does not say how often this exact combination of differences occurs.\n\n## 3. Comparability is a property of the reference, not of the number\n\nVIM §2.46 defines metrological comparability of measurement results through traceability to a common reference, as comparability of results, for quantities of a given kind, “that are metrologically traceable to the same reference” (JCGM 200:2012). Its §2.41 note permits a measurement procedure to serve as that reference for a non-ordinal quantity. The AI-Eval profile turns that into a narrower operational rule: a bridge is applicable only when the frames meet its exact aspect, scope and procedure constraints, and `mmlu` alone does not meet them.\n\nMetrological compatibility, §2.47, is a separate VIM term: it concerns whether the difference between two results is small relative to the uncertainty of that difference. It does not establish that two results share a reference.\n\nFields outside AI have already paid for this. HbA1c has two reference methods, and NGSP reports IFCC results running 1.5 to 2% HbA1c lower across the range ([NGSP](http://www.ngsp.org/ifccngsp.asp) ); in 2007 the IFCC value was given its own unit, mmol/mol, which NGSP says avoids confusion between the two systems. Cloud availability has the same shape: AWS EC2 derives Monthly Uptime Percentage from minutes of Unavailability, defined as loss of external connectivity ([AWS EC2 SLA](https://aws.amazon.com/compute/sla/) ), while Google Compute counts loss of persistent disk access as Downtime and does not count intermittent downtime shorter than one minute ([Google Compute Engine SLA](https://cloud.google.com/compute/sla) ). Two contracts print “99.99%” over different measurands.\n\nThe AI-specific documents arrive at the same statement without the metrology words. NIST’s draft practices for automated benchmark evaluations tie comparability to consistency of protocol between systems (§2.1) and describe breaking changes under semantic versioning as points after which results on either side are no longer properly comparable ([NIST AI 800-2 ipd](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.800-2.ipd.pdf) , January 2026). Practice 2.3 lists grouping together the evaluation logs that are meant to be compared as an emerging practice, and Practice 3.2 notes that an interoperable schema for sharing evaluation details may ease replication. The NAAIMES network best practice of July 2026 splits the property in two, the evaluation of outputs (scoring criteria, rubrics, graders) and the generation of outputs (prompt format, inference settings), holds that the first should always be comparable, and treats the second as depending on the goals of the evaluation (§C.1.1).\n\n`grader_id` records the grading procedure, while `prompt_protocol` records part of output generation; the two frames differ in both of those fields and in `runner_id`. An [earlier post here](/blog/ai-agent-audit-trail) put the general form in one line — a trail that records “accuracy 0.78” without the frame records a number, not an observation. The two MMLU scores are that line’s specific case with the hashes filled in.\n\n## 4. Binding the number to its frame\n\nUnder [APL](/apl), the frame is not documentation attached to a claim but a separate content-addressed object, and the claim carries only its hash. The AI-Eval profile — the vocabulary for benchmark observations about a model build — fixes what that object must contain. Frame A, indented for reading:\n\n```\n{\n  \"version\": \"0.1\",\n  \"observer\": { \"id\": \"acme-eval-lab\" },\n  \"procedure\": {\n    \"runner_id\": \"[email protected]\",\n    \"grader_id\": \"exact-match-v1\",\n    \"prompt_protocol\": \"zero-shot-mcq-v1\"\n  },\n  \"aspect\": [\"accuracy\"],\n  \"scope\": {\n    \"benchmark_id\": \"mmlu\",\n    \"benchmark_variant\": \"default\",\n    \"dataset_split\": \"dev\",\n    \"subset\": \"all\"\n  },\n  \"invariance\": [\"score-object-serialization\"],\n  \"exclusions\": [\n    \"no-production-readiness-claim\",\n    \"no-deployment-safety-claim\",\n    \"no-out-of-scope-generalization-claim\"\n  ]\n}\n```\n\nIts hash is `sha256:c7b88426f2676f3653db0fad0bdbd689318f16d589d14a315bdd4cc454bca1ab`, and the indentation above is display only: the hash is SHA-256 over the RFC 8785 canonical bytes (`src/core/jcs.rs:167`, `canonical_hash`), so whitespace never reaches the digest. Frame B is the same object with `runner_id` at [\\[email protected\\]](/cdn-cgi/l/email-protection)`grader_id` at `llm-judge-v3`, `dataset_split` at `test-lite` and no `subset` key, hashing to `sha256:c93a9c55422ddbd2158a5336caa3a251641cf3937f451fb13387a5f54f0d998e`.\n\nThe profile validator turns those fields from convention into a requirement. `procedure` must be an object with a non-empty `runner_id` (`frame.rs:35`) and `grader_id` (`frame.rs:36`); `scope` must carry `benchmark_id`, `benchmark_variant` and `dataset_split` (`frame.rs:48–50`); `aspect` must hold exactly one entry from a closed list (`frame.rs:51–57`); `exclusions` must contain all three markers, which is how a frame states what it is not evidence of (`frame.rs:69–76`). On the claim side, `content.benchmark_id` is required (`claim.rs:265`) and must equal `frame.scope.benchmark_id` (`claim.rs:106–121`), and `content.metric_id` is required and constrained by the aspect — `accuracy` admits `accuracy`, `exact-match` or `f1` (`claim.rs:268`, `315–318`).\n\nThe distance between a description and a validator is visible in the project’s own history. The design document that introduced this pair writes `procedure` as a human-readable string, `invariance` as “score object serialization” with spaces, the exclusions as two sentences of prose, the claim with `content.benchmark` and no `metric_id`, and the hashes as placeholders. The document’s examples use fields the current crate rejects. A prose value has no controlled vocabulary; different strings serialize differently and cannot satisfy exact equality.\n\nA subtler case sits inside the two frames. Frame A writes `subset: \"all\"`; frame B omits the key. A reader might expect the comparison to normalize that, an absent optional field standing for its default. It does not. Scope equality is decided by canonical bytes (`bridge.rs:114–118` calling `canonical_equal`, implemented at `src/core/jcs.rs:84–87` as `canonical_bytes(a) == canonical_bytes(b)`), and serde’s derived `PartialEq` is explicitly not used for it. Under that rule “all” and absent are two scopes. The rule leaves defaulting to the tool that emits the frame rather than to the comparison step.\n\n## 5. What the verifier says\n\n`apl-ai-eval` is a Rust crate, version 0.3.1, Apache-2.0. Given a receipt or a pair of receipts and a relation query, it decides whether the claims are well-formed under the profile and whether the relation is evaluable. The case in this post lives in its test vectors as `two_mmlu_scores_incomparable_with_ai_eval_profile`, with the pair in `test_data/vectors/pairwise/incomparable-no-bridge.json` and two sibling vector files containing bridge cases. `cargo test -p apl-ai-eval` runs 109 unit tests, 3 vector integration tests and 2 doc-tests; all 114 are green.\n\nVerify claim A on its own against the AI-Eval profile and the output is:\n\n```\n{\"core_outcome\":\"apl-valid\",\"relation_outcome\":\"relation-not-evaluated\",\"failure_classes\":[],\"diagnostics\":[\"carrier-valid\",\"apl-present\",\"apl-frame-bound\",\"apl-valid\",\"same-frame\",\"transformation-missing\"]}\n```\n\nClaim B alone produces the same shape. `apl-valid` here means well-formed and frame-bound, and says nothing about the accuracy of `0.781`; `relation-not-evaluated` is the answer to a question that was not asked.\n\nNow the pair, with the `score-delta` query from Section 1 and no bridge:\n\n```\n{\"left\":{\"core_outcome\":\"apl-valid\",\"failure_classes\":[]},\"right\":{\"core_outcome\":\"apl-valid\",\"failure_classes\":[]},\"relation_outcome\":\"incomparable\",\"diagnostics\":[\"apl-cross-frame\",\"apl-bridge-not-found\"]}\n```\n\nBoth claims have `core_outcome: \"apl-valid\"` and empty failure classes; the relation outcome is `incomparable`. `apl-cross-frame` states that the two `frame_ref` hashes differ; `apl-bridge-not-found`, that nothing was supplied to license a comparison across that difference. The two axes are separate by design: this is not a broken record and not a passing comparison. It is a subtraction with no defined result, reported as such.\n\nThe third vector is a separate adversarial case, not a bridge between Frames A and B above. Its source frame declares aspect `accuracy` and its target frame declares aspect `judge-score`. The bridge is itself a content-addressed object, and this one is well-formed:\n\n```\n{\"version\":\"0.1\",\"bridge_kind\":\"runner-equivalence\",\"source_frame\":{\"hash\":\"<FRAME_HASH:0>\"},\"target_frame\":{\"hash\":\"<FRAME_HASH:1>\"},\"comparison_scope\":{\"source_aspects\":[\"accuracy\"],\"target_aspects\":[\"judge-score\"],\"relation_type\":\"score-delta\"},\"assumptions\":[\"all MMLU results are interchangeable\"],\"losses\":[]}\n```\n\nThe `<FRAME_HASH:n>` tokens are placeholders in the recorded vector; the harness substitutes the two frame hashes at load time. The bridge parses, points at the right two frames, declares an empty `losses` array and states its assumption in plain text. The verifier’s answer:\n\n```\n{\"left\":{\"core_outcome\":\"apl-valid\",\"failure_classes\":[]},\"right\":{\"core_outcome\":\"apl-valid\",\"failure_classes\":[]},\"relation_outcome\":\"incomparable\",\"diagnostics\":[\"apl-cross-frame\",\"apl-ai-eval-bridge-aspect-family-mismatch\"]}\n```\n\nThe reported diagnostic is `apl-ai-eval-bridge-aspect-family-mismatch`. The bridge’s `comparison_scope` declares `accuracy` on the source side and `judge-score` on the target side, matching the two frames; under the profile a bridge may translate between conditions within one aspect family and may not join frames that measure different aspects.\n\nThe fourth vector is a bridge that works — same grader `exact-match-v1` on both sides, same scope `mmlu` / `default` / `dev` / `all`, only `runner_id` differing:\n\n```\n{\"left\":{\"core_outcome\":\"apl-valid\",\"failure_classes\":[]},\"right\":{\"core_outcome\":\"apl-valid\",\"failure_classes\":[]},\"relation_outcome\":\"bridged-comparable\",\"diagnostics\":[\"apl-cross-frame\",\"apl-bridge-applicable\"]}\n```\n\n`bridged-comparable` indicates that the frames differ and that an applicable bridge, with its stated assumptions and losses, was supplied.\n\nThe profile permits a `repeatability` bridge only for the `repeatability-check` relation type; it does not license `score-delta`. Repeatability of a procedure says nothing about a difference between two subjects measured under it. The code pins repeatability bridges to the `repeatability-check` relation type, rejecting anything else with `apl-ai-eval-relation-type-invalid` (`bridge.rs:167–172`), and confines runner- and grader-equivalence bridges to `score-delta` (`bridge.rs:129–164`).\n\nMalformed records sit on a separate axis. A claim whose subject lacks `type` and `artifact_digest`, whose `procedure` and `scope` are strings rather than objects, and whose exclusions are incomplete returns `apl-invalid`, `failure_classes: [\"claim-structure-failure\"]`, diagnostic `apl-subject-invalid`. Invalid means the record is not a well-formed observation; incomparable means two well-formed observations cannot be subtracted.\n\nThe crate is at [apl-protocol.org/implementations/apl-ai-eval](https://apl-protocol.org/implementations/apl-ai-eval) ; the source, the vectors and the three pairwise JSON files quoted above are at [github.com/evidentum-io/apl-core/tree/main/apl-ai-eval](https://github.com/evidentum-io/apl-core/tree/main/apl-ai-eval) .\n\n## 6. What this does not prove\n\nThe frame settles a narrow question and leaves the rest open.\n\n- **That either score is correct.**`apl-valid` is a statement about structure. MMLU-Redux re-annotated 5,700 questions and estimates “that 6.49% of MMLU questions contain errors” ([arXiv:2406.04127](https://arxiv.org/abs/2406.04127)  ). A frame binds a number to its declared conditions; it does not establish that those conditions were executed as declared.\n- **That two identical identifiers name the same thing.** Byte equality knows that[\\[email protected\\]](/cdn-cgi/l/email-protection)\n[\\[email protected\\]](/cdn-cgi/l/email-protection)\n- **That the dataset was the same dataset.** The frame identifies data by name and split, not by digest. The 14079-against-14042 gap in Section 2.1 confirms an open question left in the earlier post: if a name resolves to two different sets of questions, whether a dataset digest should be recorded remains an open design question.\n- **Anything about thresholds, or when they were set.** A frame records the procedure, not a release threshold or when it was chosen. NIST’s Practice 1.2 addresses documenting what an evaluation measures before conducting it.\n- **Anything about uncertainty.** Profile v0.1 specifies no uncertainty field, and neither claim records`sample_count` . For illustration, a proportion estimated from 285 independent items has a wider binomial interval than one estimated from 14,042 items; that comparison does not establish the uncertainty of either displayed claim. ISO/IEC 17025:2017, 7.8.3.1 c), calls for reporting measurement uncertainty where relevant. AI-Eval v0.1 specifies no field for it.\n- **That the split labels mean what a reader assumes.**`dev` is a real MMLU split and, by the paper’s design, the source of its few-shot examples; scoring on it is a decision the frame makes visible and does not defend.`test-lite` is the label in the crate’s test vectors and names no published artifact.\n- **That the vocabulary is stable.** AI-Eval is version 0.1, a draft. NIST describes a breaking change in an evaluation as a point after which results on either side are no longer properly comparable.\n\nWhat the frame establishes is narrower: it binds a stated procedure to a stated claim by hash. Under AI-Eval v0.1, the supplied frames and this `score-delta` query do not satisfy the conditions for an applicable bridge, so the verifier returns `incomparable` rather than a delta.\n\n*If you are building evaluation infrastructure where the numbers will be compared across teams, vendors or quarters, and the comparison has to survive being questioned, that is [contract work I take on](/hireme).*", "url": "https://wpnews.pro/news/the-two-mmlu-scores-what-a-benchmark-name-does-not-fix", "canonical_source": "https://zatona.dev/blog/the-two-mmlu-scores", "published_at": "2026-09-14 03:41:07+00:00", "updated_at": "2026-09-14 03:56:47.348934+00:00", "lang": "en", "topics": ["ai-research", "machine-learning", "ai-safety"], "entities": ["MMLU", "Dmitrii Zatona", "apl-ai-eval", "acme-gpt-7b", "APL AI-Eval", "RFC 8785"], "alternates": {"html": "https://wpnews.pro/news/the-two-mmlu-scores-what-a-benchmark-name-does-not-fix", "markdown": "https://wpnews.pro/news/the-two-mmlu-scores-what-a-benchmark-name-does-not-fix.md", "text": "https://wpnews.pro/news/the-two-mmlu-scores-what-a-benchmark-name-does-not-fix.txt", "jsonld": "https://wpnews.pro/news/the-two-mmlu-scores-what-a-benchmark-name-does-not-fix.jsonld"}}