{"slug": "making-flawed-benchmarks-useful", "title": "Making Flawed Benchmarks Useful", "summary": "Blatent found that across six established evaluation suites, models that extracted correct answers from source text still lost credit when their wording differed from the reference, including a case where a model answering \"CNN\" instead of \"CNN's\" to a question about Larry King's employer scored zero on both normalized exact match and token-overlap F1. The company, which was evaluating extraction accuracy for its Struct product, cited Epoch AI's benchmark reviews and a paper on physics benchmarks (arXiv 2609.13009) showing a PHYBench grader awarding zero to an algebraically equivalent expression, and argued that neither exact match nor token F1 understands what an answer means.", "body_md": "# Making flawed benchmarks useful\n\nWe set out to measure Struct's extraction accuracy. Along the way, we discovered that sometimes the answers were more reliable than the scores.\n\nAt Blatent, we approach model development scientifically, and with a Bayesian instinct: start with our best understanding, gather evidence, update our beliefs and iterate. But how much weight we give that evidence depends on how it was produced. Whether a result confirms our expectations or challenges them, we need to understand what the experiment actually measured.\n\nThis post follows our early work evaluating extraction: discovering the richness of open benchmark data, the noise it can contain, and the care needed to turn it into useful evidence about our models.\n\nA benchmark score looks reassuringly simple. Give several models the same questions, count the correct answers, and compare the results.\n\nThe difficult part is deciding what “correct” means.\n\nEpoch AI’s benchmark reviews illustrate how much can go wrong. Its [review of SWE-bench Verified](https://epoch.ai/benchmarks/swe-bench-verified/review) describes tests that reject functionally correct solutions. Its [broader review framework](https://epoch.ai/data/benchmark-reviews-documentation/methodology) covers the opposite problem too: scorers that reward shortcuts or accept incorrect answers. Both can distort our understanding of what a model can do.\n\nSome of us at Blatent have backgrounds in physics, so a recent [paper examining physics benchmarks](https://arxiv.org/pdf/2609.13009) caught our attention. One example, from PHYBench, asks for the tension in a rope holding a group of spheres together. The model gave the expression on the left; the reference used the one on the right:\n\nIf you've done a bit of algebra, you'll immediately see that these are exactly the same expression, just written differently. Yet the grader awarded the model zero points.\n\nWe encountered a less mathematical version of the same problem while evaluating exact-source extraction for Struct. The task is simple: give a model a passage of text, ask it a question, and have it extract the answer directly from the source, preserving the original wording. No paraphrasing or generating an answer from scratch. The extracted answer is then compared against a reference provided by the benchmark.\n\nAcross six established evaluation suites, we found examples where models extracted the right information but lost credit because their answers differed from the reference wording.\n\nSome differences were almost comically small.\n\nA question asked which company Larry King worked for. The reference answer was:\n\nCNN’s\n\nThe model answered:\n\nCNN\n\nIt received zero for exact match, and zero under the benchmark’s partial-credit metric.\n\nThe company was correct. The missing possessive suffix decided the score.\n\n## How partial credit still gets this wrong\n\nIt would be unfair to describe these benchmarks as simply comparing raw strings. Most of the scoring procedures we used include two measures: normalized exact match and token-overlap F1.\n\nExact match first cleans up the text. It ignores capitalization, removes punctuation and English articles, and normalizes whitespace. This prevents harmless differences such as “the answer” versus “Answer” from becoming errors.\n\nToken F1 goes further, awarding partial credit for words shared between the prediction and a reference answer. If there are multiple reference answers, it uses the best match.\n\nBoth are reasonable approaches, and they solve some obvious problems. But neither actually understands what the answer means.\n\nIn the CNN example, removing punctuation turns `CNN’s`—written with a straight apostrophe in the dataset—into `cnns`. The model’s answer becomes `cnn`. Tokens are common chains of characters, not simply alphabetical characters. In this case there are no matching tokens, so even F1 awards zero.\n\nEach dataset suffered from these flaws and examples expose different versions of the same problem:\n\n| Reference answer | Model answer | Exact match | Token F1 | \n|---|---|---|---|\n| `CNN's` | `CNN` | 0% | 0% | \n| `programmed death receptor-1` | `PD-1` | 0% | 0% | \n| `60 euros` | `$89` | 0% | 0% | \n\nIn the second example, the passage explicitly introduces “programmed death receptor-1 (PD-1).” The equivalence is supplied by the source itself.\n\nIn the third, the question asks what dog shoes cost. The passage gives the price as “60 euros -- $89.” The reference keeps the euro amount; the model extracts the dollar amount. Both come directly from the source, but they share no tokens after normalization, so both metrics award zero.\n\nThese examples affected Struct and the comparison models, such as Clairic, GPT-5.6 Terra and Luna, Claude’s Sonnet and Haiku and the various flavours of Gemini. They reveal limitations in the measurement that apply regardless of whose model produced the answer.\n\nThe dataset versions matter here. Our six suites included the [MRQA adaptations](https://github.com/mrqa/MRQA-Shared-Task-2019) of Natural Questions, NewsQA and BioASQ, alongside AdversarialQA, TyDiQA-GoldP and an extraction-compatible subset of QASPER. Original BioASQ, for example, supports accepted synonyms and uses different scoring procedures. The examples above concern the particular datasets and evaluators we ran.\n\n## When a more explicit answer scores worse\n\nThings get even more interesting when a model doesn't get zero, but actually loses points for giving a more complete answer.\n\nWe came across a particularly good example in QASPER, a benchmark built around questions about scientific papers.\n\nThe question was:\n\nWhat is the size of the dataset?\n\nThe source described a collection of “over 30,000 images with 5 crowdsourced descriptions each.”\n\nTwo annotators supplied these reference answers:\n\n- `30,000`\n- `collection of over 30,000 images with 5 crowdsourced descriptions each`\n\nBoth identify the relevant information, but they choose very different answer boundaries.\n\nClairic answered `over 30,000` and received 66.7% Answer F1.\n\nStruct answered `over 30,000 images` and received 50%.\n\nStruct included the unit, making the answer more informative, and lost points for it.\n\nQASPER’s [official answer metric](https://github.com/allenai/qasper-led-baseline/blob/main/scripts/evaluator.py) is token F1, so it would be wrong to say these answers received no credit. The issue is subtler: the amount of credit does not reliably track how well the answer communicates the requested information.\n\nThis limitation is also well established. The [original NewsQA paper](https://arxiv.org/html/1611.09830) explicitly discusses semantically equivalent answers receiving different scores. The benchmark’s authors recognized that automatic lexical metrics could miss meaningful equivalence.\n\n## The question can be useful even when the score is misleading\n\nA benchmark contains several things that are easy to conflate:\n\nThe source provides information. The question specifies what to find. The reference records an accepted answer. The scorer decides how other answers compare with that reference.\n\nEach can have a different quality problem.\n\nA question might be ambiguous. A reference might be factually wrong. A scorer might reject a valid abbreviation. These problems require different remedies.\n\nThe extraction examples above often leave the underlying task intact. There is still a useful question to answer and enough evidence to answer it. What needs improvement is the way alternative answers are assessed.\n\nA reference answer is evidence of what counts as correct. It is not always an exhaustive specification of every correct response.\n\nThat distinction matters when using benchmarks to choose a model. A high lexical score can reflect both successful information extraction and successful reproduction of annotation conventions. The score alone does not tell us how much each contributed.\n\n## Looking beyond word overlap\n\nWe didn't want to discard useful benchmarks just because their scoring was imperfect. The questions and source material were still valuable. What we needed was a better way to assess the answers.\n\nWe approached this in two steps.\n\nFirst, we check whether the model's answer actually appears in the original source text. This is a deterministic check, and an important one: not every model guarantees exact-source extraction. If a model paraphrases an answer, even correctly, it hasn't completed the task we asked it to perform.\n\nOnly then do we evaluate whether the extracted answer is actually correct. Manually reviewing thousands of responses wasn't practical, so we use SOTA model, OpenAI's Astra, to judge whether each answer correctly addresses the question in the context of the original source.\n\nThis allows us to recognise things that token matching misses: that PD-1 and programmed death receptor-1 refer to the same thing, that “60 euros” and “$89” are equivalent prices in this passage, or that adding the word images doesn't make an otherwise correct answer worse.\n\nWe also preserve the original benchmark scores and examine cases where they disagree with Astra's judgments. This helps us understand whether a difference comes from the model's answer or from how the benchmark scores it.\n\nOf course, there's a catch. We've just spent several sections showing how graders can get things wrong, and an LLM acting as a grader is no exception. Astra can misunderstand a passage, accept an unsupported answer or reject a correct one.\n\nBut by checking source fidelity separately from answer correctness, we can get a more useful picture of how models actually perform on extraction tasks.\n\n## When the scorers disagree\n\n### what company does larry king work for?\n\n- \"CNN's\"\n\n| Model / answer | Exact match | Token F1 | Astra | \n|---|---|---|---|\n| Struct\"CNN\" | 0% | 0% | Accepted | \n| Cloudflare Clairic\"CNN's\" | 100% | 100% | Accepted | \n\nin an interview Tuesday on CNN's \"Larry King Live.\"\n\nCNN is the company identified by the passage as airing Larry King's show.\n\n## Keeping the useful parts\n\nOur aim in developing Struct is practical usefulness. In this evaluation, that means reliably extracting the information a user needs from the documents. A benchmark score matters insofar as it helps us understand and improve that capability.\n\nOpen benchmark datasets have been valuable for this. They provide varied source material, challenging questions and human annotations that would take considerable effort to assemble ourselves. Our experience showed that we could learn more from them by being precise about which parts deserved our trust. A useful question can have an incomplete set of reference answers. A correct reference can sit behind a scorer that rejects equally valid responses.\n\nThe challenge is distinguishing genuine model improvements from changes that simply make a model better at matching a benchmark's expectations. Otherwise, we risk making a model look better on a test without making it more useful to anyone.\n\nThis brings us back to the experimental approach we started with. Updating our beliefs includes revisiting the measurement itself. Sometimes the evidence tells us to improve the model. Sometimes it tells us to improve the experiment. The purpose of evaluation is to help us tell the difference and build something more useful as a result.", "url": "https://wpnews.pro/news/making-flawed-benchmarks-useful", "canonical_source": "https://blatent.ai/blog/making-flawed-benchmarks-useful", "published_at": "2026-11-04 18:02:44+00:00", "updated_at": "2026-11-04 18:20:42.646438+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "artificial-intelligence"], "entities": ["Blatent", "Struct", "Epoch AI", "SWE-bench Verified", "PHYBench", "Larry King", "CNN"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/making-flawed-benchmarks-useful", "markdown": "https://wpnews.pro/news/making-flawed-benchmarks-useful.md", "text": "https://wpnews.pro/news/making-flawed-benchmarks-useful.txt", "jsonld": "https://wpnews.pro/news/making-flawed-benchmarks-useful.jsonld"}}