# The Benchmark > Source: > Published: 2026-07-19 00:00:00+00:00 A benchmark score is a manufactured number. A group chooses tasks, samples items, formats prompts, runs a model version, applies a scorer, handles failures, aggregates results, and publishes a statistic. Each operation leaves a provenance record. Without that record, a leaderboard row cannot tell a buyer whether the model will answer support tickets, repair a repository, summarize clinical notes, or satisfy a latency budget. This drawing traces three instruments through the same measurement pipeline. MMLU uses fixed-answer multiple-choice questions across academic subjects [1]. SWE-bench asks a system to modify real software repositories and lets tests judge the patch . Chatbot Arena gathers pairwise human preferences between anonymous model responses [[2]](#ref-jimenez-2024-swebench). Their numbers answer different questions because their samples, interfaces, scorers, and estimands differ. [[3]](#ref-chiang-2024-arena)The useful question is therefore specific: does this evidence reduce uncertainty about the decision in front of us? A valid evaluation connects the intended use to representative tasks and consequences. Ranking is a downstream convenience. Provenance, uncertainty, slices, contamination checks, and operating cost determine whether the rank deserves any weight. ## Every score passes through a chain of choices Begin at the right edge: a decision. A technical PM may choose a model for a customer-support workflow. An engineering leader may fund an agent roadmap. A journalist may assess a vendor claim. Work backward from that choice to the capability and constraints that matter, then select evidence that represents them. Reversing this order—starting with an available leaderboard—quietly lets the benchmark define the problem. The pipeline begins with a target population such as “production bug reports in maintained Python services.” A sampling frame makes that population accessible. Inclusion rules, time windows, languages, repositories, topics, and difficulty filters produce items. A prompt adapter turns each item into model input. The run records checkpoint or API version, tools, temperature, token budget, retries, and dates. The scorer converts outputs into item results. Aggregation turns those results into a headline and intervals. A report maps the evidence to a decision. Each arrow can change the final number while model weights stay fixed. Reword choices, allow a tool, normalize punctuation, count timeouts as zero, discard malformed items, weight subjects equally, or weight every item equally: each creates a different estimand. Provenance turns those choices into inspectable facts. Version the whole chain as one evaluation release. A model name alone is insufficient for an API that changes behind an alias; record the provider’s dated model identifier and retain raw responses when terms permit. Hash item files, prompt templates, scorer code, and environment manifests. Give every run an ID that joins those artifacts to item results and invoices. This turns a score from a slide annotation into an auditable dataset. A later team can reproduce the aggregate, identify why two reports differ, and decide whether a rerun is comparable. Missingness needs equal care. An item can disappear because a parser rejected output, a sandbox failed, a request timed out, a safety system refused, or an evaluator excluded it after inspection. Each cause carries information about deployment behavior. Set policies before the run, report counts by cause, and include failures in the denominator unless the declared estimand truly excludes them. Selective denominators can raise a score while the useful completion rate falls. ## Sampling and prompt format define what receives a chance to succeed A benchmark sample stands in for a larger population. MMLU assembled 15,908 multiple-choice questions from 57 subjects, ranging from elementary mathematics to professional law. Its subject inventory broadens coverage, yet it remains a collection of written exam questions with four supplied choices. It measures performance under that interface, not open-ended work across every domain represented by the subject names. Sampling error begins before a confidence interval. A convenient collection may omit the hardest customers, private codebases, recent regulations, uncommon languages, long contexts, or cases where abstention is correct. Deduplication can remove near copies while leaving template families concentrated. A stratified sample can guarantee enough high-cost or low-frequency cases to estimate them, then use production prevalence when computing the deployment estimate. Prompt format is part of the instrument. Hendrycks and colleagues reported five-shot GPT-3 [4] accuracy of 43.9% on MMLU in the benchmark’s September 2020 paper; the number is tied to that model, prompt regime, item set, and publication date. Changing instructions, answer extraction, choice order, examples, or chain-of-thought policy can change the measured score without changing the underlying checkpoint. A benchmark report should publish the rendered prompts or a deterministic adapter that recreates them. Prompt sensitivity is diagnostic. If harmless formatting moves a score enough to change a purchase, measure that variation as part of the system. Production includes templates, parsers, retrieval, and tool policies; evaluating the naked model may answer a research question while missing the deployed unit. Representative sampling also depends on time. Product traffic, code dependencies, regulations, and user vocabulary shift. A random sample from last year may be statistically precise for a population that no longer exists. Stamp the collection window, measure drift in task and outcome frequencies, and schedule refreshes. Preserve old sets for longitudinal comparison, but do not let continuity overrule current validity. A stable score on stale work can coexist with declining production utility. ## Fixed answers, executable tasks, and preferences observe different evidence The three instruments share a pipeline and diverge at the task contract. MMLU presents a question with four choices and scores whether the extracted choice matches the key. Its advantages are low scoring cost, repeatability, and broad subject labels. Its failure lines include guessing, answer-key errors, prompt sensitivity, and weak correspondence to open-ended practice. SWE-bench starts from a GitHub issue and a repository snapshot. A system must generate a patch, after which repository tests determine resolution. The original October 2023 paper reported that its best evaluated model resolved 1.96% of tasks. That dated result described the paper’s harness, selected systems, repository versions, and task set; later leaderboard numbers use changed agents, tools, budgets, and benchmark variants and require their own provenance. Executability strengthens the oracle, while flaky tests, environment failures, underspecified issues, and patch leakage remain measurement risks. Chatbot Arena presents a user prompt to two anonymous systems and asks a person which response is better or whether they tie. The March 2024 paper analyzed more than 240,000 votes and used a Bradley–Terry model [5] for ranking. Preference can capture usefulness and style that exact matching misses. It also mixes user populations, prompt distributions, position effects, response lengths, model availability, and evolving traffic. The score estimates comparative preference under that arena, not factual accuracy in isolation. No conversion makes these three scores interchangeable. A fixed key observes selection under an exam format. Tests observe whether code changes satisfy executable conditions. Votes observe relative human preference for displayed responses. Choose the instrument whose evidence resembles the consequential part of the decision. Scorer quality deserves its own evaluation set. Seed known correct, partially correct, harmful, and irrelevant outputs; include adversarial formatting and plausible falsehoods. Measure human agreement, test flakiness, judge bias, and calibration by slice. Freeze scorer versions for a comparison. When a judge or test suite changes, label the result as a new series rather than joining incompatible scores on one trend line. ## Aggregation compresses failures and can reverse a ranking An item scorer produces data: exact correctness, tests passed, a rubric vector, a preference, latency, cost, or several of these. Aggregation chooses what counts equally. A micro-average weights every item equally. A macro-average can weight subject or customer slices equally. Utility weighting values outcomes by business frequency or consequence. These are policy choices expressed as arithmetic. Consider two systems across knowledge, code, and service tasks. Model A scores 92, 46, and 40; Model B scores 78, 76, and 72 in an illustrative fixed dataset. Equal slice weights put B ahead. A portfolio dominated by knowledge questions can put A ahead. Neither calculation is fraudulent. Each estimates a different target mixture. Publishing only the winning mean hides the assumption that selected the winner. Means also conceal tail failures. A support model can average 90% while failing nearly every cancellation request. A coding agent can resolve routine dependency bumps while corrupting migrations. Report the aggregate beside slice counts, intervals, severity, abstentions, timeouts, and examples [6]. A small severe slice may warrant a release block even when its population-weighted contribution barely moves the mean. Simpson’s paradox supplies a sharper warning. A candidate can perform better within every meaningful slice yet appear worse overall if candidates are evaluated on different slice mixtures. Paired evaluation on the same items prevents that particular comparison error. If traffic allocation differs, standardize both systems to one declared population before ranking them. Always publish the count and weight beside a slice score; percentages without denominators hide how much evidence each bar contains. **1. MA** 83.7 ± 7.1 **2. MB** 78.9 ± 7.8 **3. MC** 74.1 ± 8.3 Change the scorer or task mix. Reduce the sample to widen intervals. The contaminated toggle gives Model A an artificial lift. The demo holds prepared item-level values fixed. Exact match, unit-test, pairwise-preference, and rubric-judge modes impose different score mappings. Task weights change the target population. Judge noise moves subjective modes. Sample size changes interval width. The contamination toggle grants one candidate an artificial advantage. The ranking reorders because measurement choices have changed, not because the model learned during the interaction. For [RAG stage metrics](/blog/the-rag-pipeline), retrieval recall and answer faithfulness belong on separate lines before any composite. The existing [RAG evaluation guide](/blog/rag-evaluation) applies that separation to a specific system. The same rule generalizes: keep diagnostic measures visible beneath a decision metric. ## A gap smaller than measurement uncertainty cannot carry a rank A score computed on a sample is an estimate. If binary item outcomes were independent draws from one population, a rough standard error would be `√[p(1−p)/n]` . A 95% normal interval uses about 1.96 standard errors. Near 50%, 100 independent items yield a margin near ten percentage points; 1,000 yield about three. Small benchmark subsets cannot support fine-grained rank claims. Real evaluations violate the simple assumptions. Items share templates and sources. Multiple questions come from one repository. Repeated model samples share prompts. Human votes cluster by user and time. Judge models can have systematic preferences. Bootstrap by the true sampling unit, cluster related observations, repeat stochastic runs, and state which sources of variation the interval includes. Pairwise systems add another dependency: ratings are relative to the opponents and traffic observed. Elo is familiar, while Bradley–Terry modeling estimates latent strengths from win probabilities; both depend on comparison structure. A model can gain or lose rank when competitors or prompt mixes change. Publish vote counts, pairing policy, interval method, and snapshot date. “Rank 3” without them suggests precision the experiment may not contain. Practical significance comes after statistical uncertainty. A one-point quality difference may be irrelevant beside a fivefold cost difference, or decisive in a high-volume safety slice. Set the smallest decision-relevant difference before looking at results. If intervals cross that threshold, gather more representative data or accept that the evidence does not distinguish candidates. Multiple comparisons make leaderboards especially prone to lucky winners. If dozens of models, prompts, and checkpoints are tried against one set, the highest observed score benefits from selection noise. Report the search process, keep a final untouched set, and adjust claims when many hypotheses were tested. A confidence interval computed after choosing the winner does not include the uncertainty created by that choice. Replication on fresh items is stronger evidence than another decimal place on the development set. Uncertainty should cover operational randomness when the decision includes it. Retry policies, temperature, retrieval updates, tool failures, and human queues can alter outcomes. Run enough repeats to estimate within-item variance, then separate it from between-item variation. Paired differences often need fewer items than independent comparisons because both candidates face the same tasks. Archive seeds where supported, while remembering that hosted systems can remain nondeterministic under identical requests. ## Exposure to test material can inflate a score without improving transfer Contamination occurs when evaluation information influences model training, selection, prompting, or tool setup in a way unavailable in the target use. Direct train/test leakage is the clearest case: benchmark questions or solutions enter pretraining or fine-tuning [7]. Near duplicates, translated copies, answer explanations, repository patches, and benchmark-specific tutorials create less visible routes. Repeated tuning against a public leaderboard leaks test feedback into development even when raw items never enter training. The effect depends on the instrument. A fixed-answer item may be memorized verbatim. An executable repository task may leak through the merged patch or issue discussion. A preference benchmark with fresh live prompts reduces static-item memorization but can still be gamed through style tuning, model identity cues, or repeated optimization against arena users. Private tests and held-out prompts slow adaptation; they do not establish permanent purity. A contamination register records item source, public date, suspected exposure route, similarity evidence, affected model versions, risk, and action. Deduplicate with exact hashes and semantic similarity, inspect suspiciously distinctive strings, use canaries where lawful, compare performance by item age, and retain a post-cutoff set. None proves absence. Training corpora are often undisclosed, and semantic knowledge can legitimately transfer. Interpret a suspected lift by triangulation. Does it concentrate on old public items? Does performance collapse on paraphrases or newly authored analogues? Can the model reproduce answer-specific artifacts? Does a clean temporal holdout preserve the gain? Report uncertainty around the diagnosis. “Contamination detected” needs an exposure definition and evidence, while “no contamination” usually exceeds what the audit can establish. Contamination and legitimate learning can look similar. Reading public documentation before solving a task is expected in many jobs; memorizing the private acceptance answer is not. Define what information the deployed system would be allowed to access and judge exposure against that boundary. For an open-book workflow, web access may improve validity. For a closed-book knowledge probe, the same access invalidates the construct. The rule follows the decision, not a universal ban on prior exposure. Temporal splits help when model-training and item-publication dates are credible. Put post-cutoff issues or questions in a clean partition and compare them with older matched items. Control for topic and difficulty so recency does not become the explanation. For repository tasks, retain issue, base commit, environment, tests, and patch timestamps. A date attached only to the leaderboard snapshot cannot establish that test information postdated training. ## Optimization pressure turns measurement shortcuts into product strategy Goodhart’s law describes what happens when a measure becomes a target: the relationship between the measure and the intended goal weakens under optimization [8]. A benchmark rewards whatever reliably raises its score. Those features may include genuine capability, but they can also include answer-format tricks, verbosity preferred by judges, benchmark-specific prompting, test exploitation, selective submission, or spending far more inference compute than a production budget permits. Gaming does not require fraud. Teams rationally optimize the published acceptance criterion. If a leaderboard omits cost, a search-heavy agent can buy quality with tokens. If only the best of many runs is submitted, variance becomes an advantage. If aggregate preference dominates, pleasing style can offset factual or safety regressions. If a benchmark is static, development gradually specializes to its item families. Defenses distribute pressure. Keep a final holdout unavailable during development. Rotate fresh items. Require full run logs rather than cherry-picked attempts. Report cost, latency, tool use, and retries with quality. Use multiple measures that fail differently. Audit surprising gains. Test transfer on a separate task family. Retire compromised items while preserving versioned historical results. Benchmark saturation can mean progress, leakage, narrow item coverage, or inadequate resolution. Raising difficulty alone can produce puzzles unrelated to actual decisions. Refresh from the target population and preserve known failure costs. A difficult test is useful only when passing it predicts something valuable. - ▸Task specification and representative sampling plan - ▸Scorer contract and contamination register - ▸Confidence-interval prompts and failure-slice table - ▸Human-review rubric with cost and latency fields - ▸Public-leaderboard reading checklist ## Validity comes first: design the evaluation around the choice Write the decision sentence before selecting a benchmark: “Choose the system that completes these tasks for these users, under these permissions, at this quality, cost, and latency, with these failures treated as release blockers.” That sentence identifies the population, unit of analysis, operating envelope, and loss function. It also reveals where a public benchmark can contribute evidence and where local measurement is required. Build a task inventory from actual work: sampled logs with privacy controls, tickets, repository changes, analyst requests, and known incidents. Stratify by frequency and consequence. Preserve a frozen acceptance set, then maintain a diagnostic set that grows with discovered failures. Specify whether the evaluated unit is a base model, prompt, retrieval pipeline, agent, or complete service. Record every version. Choose scorers by observable success. Exact match fits normalized identifiers and constrained answers. Executable checks fit code and structured operations when tests represent the requirement. Rubrics fit qualities with multiple valid outputs, provided criteria, anchors, reviewer calibration, blinding, and disagreement handling are explicit. Pairwise preference fits comparative experience; add factual and policy checks when preference alone can reward an attractive error. Model judges reduce cost but require validation against qualified humans on the same distribution. Pre-register aggregation, exclusions, sample size, interval method, and release thresholds. Evaluate candidates on paired items so task difficulty cancels in comparisons. Count invalid outputs, crashes, and timeouts rather than silently dropping them. Report both population-weighted utility and protected high-severity slices. Add per-task cost, p50 and p95 latency, human review minutes, and retry rate. A model that wins on quality outside the operating budget has not won the product decision. Make the release rule capable of returning “no selection.” Procurement processes often force a winner even when all candidates miss the minimum quality bar or their intervals overlap. A valid evaluation can recommend keeping the current system, narrowing scope, adding human review, collecting more evidence, or postponing launch. This option reduces pressure to turn weak rank evidence into certainty. Separate selection from monitoring. The acceptance evaluation chooses whether a version may ship. Production monitoring checks drift, incidents, cost, latency, and user outcomes after release. Diagnostic tests help engineers localize failures. These sets can share task taxonomies while serving different statistical purposes. Reusing one small benchmark for every purpose exposes it to repeated optimization and makes its errors harder to interpret. Run the evaluation as a recurring measurement system. New production failures enter diagnostics, then fresh acceptance sets after review. Distribution shifts trigger resampling. Scorer drift triggers recalibration. Public leaderboards remain useful context: they can screen candidates, expose methods, and supply standardized comparisons. Local evidence completes the argument. The final report should make disagreement productive. A reader can challenge whether the sampled tasks represent use, whether a scorer captures success, whether weights match priorities, whether uncertainty permits a rank, or whether exposure compromised the test. Those are resolvable measurement questions. A bare score offers nowhere to inspect the claim. - [01] [Hendrycks et al., “Measuring Massive Multitask Language Understanding,” arXiv:2009.03300v3, 2021](https://arxiv.org/abs/2009.03300)[↩](#cite-hendrycks-2021-mmlu) - [02] [Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?,” arXiv:2310.06770v3, 2024](https://arxiv.org/abs/2310.06770)[↩](#cite-jimenez-2024-swebench) - [03] [Chiang et al., “Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference,” arXiv:2403.04132v1, 2024](https://arxiv.org/abs/2403.04132)[↩](#cite-chiang-2024-arena) - [04] [Brown et al., “Language Models are Few-Shot Learners,” arXiv:2005.14165v4, 2020](https://arxiv.org/abs/2005.14165)[↩](#cite-brown-2020-fewshot) - [05] [Bradley and Terry, “Rank Analysis of Incomplete Block Designs: The Method of Paired Comparisons,” Biometrika 39, 1952](https://academic.oup.com/biomet/article-abstract/39/3-4/324/326091)[↩](#cite-bradley-terry-1952) - [06] [Liang et al., “Holistic Evaluation of Language Models,” arXiv:2211.09110v2, 2023](https://arxiv.org/abs/2211.09110)[↩](#cite-liang-2023-helm) - [07] [Dodge et al., “Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus,” EMNLP 2021](https://aclanthology.org/2021.emnlp-main.98/)[↩](#cite-dodge-2021-c4) - [08] [Goodhart, “Problems of Monetary Management: The UK Experience,” Monetary Theory and Practice, 1984](https://link.springer.com/chapter/10.1007/978-1-349-17295-5_4)[↩](#cite-goodhart-1984) Anything on this sheet still unclear — or anything you were too polite to ask out loud? File an RFI. Answers come from the drawing itself and cite their sheet numbers, and every question is recorded in the drawing log so the next revision can answer it in print.