{"slug": "my-factual-recall-tasks-were-scoring-format-not-facts", "title": "My factual-recall tasks were scoring format, not facts", "summary": "A developer's drift-detection harness revealed that two factual-recall tasks, fact-element and fact-planet, were grading output format rather than knowledge, with Claude Sonnet 5 failing the gold-symbol task 60 out of 60 times and GPT-4o mini failing 59 out of 59 despite both models answering correctly. The near-identical control task fact-capital, which explicitly instructs models to answer with only the city name, passed cleanly across all runs, showing the failures stem from models adding punctuation or markdown rather than lacking the facts. The developer estimates the two tasks account for 5.7 points of every affected model's score and skews the public board's factual-recall category breakdown.", "body_md": "*Originally published at [erikhill.dev](https://erikhill.dev/notes/2026-09-22-my-factual-recall-tasks-were-scoring-format-not-facts/). The numbers below are checked against the repository they come from.*\n\n**This is a finding about my own harness. The suspect is the probe, not the models it measures.**\n\nI built a detector that decides whether a day's drift run contains anything worth\n\nwriting up. The first thing it did was accuse my own suite. One task,\n\n`fact-element`, had failed on three or more providers on the same day, on 46\n\nseparate days.\n\nThe task is this:\n\nWhat is the chemical symbol for gold? Two letters only.\n\nGraded by exact match against `au`, after stripping whitespace and lowercasing.\n\nClaude Sonnet 5 failed it **60 times out of 60**. GPT-4o mini failed it **59 out of 59**. Llama 3.1 8B, 23 out of 23. Those are not knowledge results. A model\n\n| claim | how it was checked | \n|---|---|\n| `fact-element` failed on 3 or more providers on the same day, 46 times | cross-provider flip analysis over every stored run in `dashboard/drift_board.json` | \n| those were graded failures, not outages | every run counted had `reliability: 1.0` , so no call errored and none was truncated | \n| the models know the answer | `gpt-4o-mini` answered`'The chemical symbol for gold is Au.'` ;`claude-sonnet-5` answered`'**Au**\\n\\nThat's the chemical symbol for gold, derived from the Latin word *aurum*.'` | \n| the same models pass a near-identical task | all three answered `fact-capital` with exactly`'Tokyo'` , 0 failures in 59 and 60 runs | \n| one trailing period is the whole difference | `gpt-4o-mini` answered`fact-planet` with`'Mercury.'` and was recorded wrong 59 times out of 59 | \n| a model that formats tersely passes cleanly | `claude-opus-4-8` answered`'Au'` ,`'Mercury'` ,`'Tokyo'` , and fails`fact-element` on 1 of 60 runs | \n\nThe answers above are not from the board. They came from nine live calls made\n\nthrough the probe's own call path, printed with `repr()` so punctuation and\n\nmarkdown are visible rather than inferred.\n\n`fact-capital` is the control, and it is the reason this is a finding rather\n\nthan a theory. It asks for the capital of Japan and every model in the table\n\nanswers `'Tokyo'`, exactly, on every run. Same grader, same strictness, same\n\nmodels, same day. Nothing about exact matching is broken in general.\n\nThe three tasks differ in one way. `fact-capital` says **\"Answer with only the city name\"**, which is an instruction about the shape of the output.\n\n`fact-element` says `fact-planet` says The suite has 35 tasks, so the smallest accuracy change it can print is\n\n100/35 = 2.86 points. `fact-element` and `fact-planet` together are 5.7 points\n\nof every affected model's score, permanently, for reasons that have nothing to\n\ndo with the models.\n\nWorse than the points: these tasks are tagged `factual-recall`, and that tag\n\nfeeds a per-category breakdown on the public board. GPT-4o mini's\n\nfactual-recall score has been reading as a knowledge number when two thirds of\n\nit is a formatting number. Anyone comparing models on that column, including\n\nme, was comparing how terse they are.\n\nThe part I am least sure of is the explanation, not the measurement. The numbers\n\nare solid and the live answers are quoted verbatim, but \"models treat *answer with only X* as a constraint and \n\nTwo things genuinely argue against my reading. Grok 4.5 fails `fact-element` on\n\n26 of 49 runs, which is close to a coin flip, and a rigid house style should not\n\nproduce that. And Claude Sonnet 5 failed `fact-planet` on 38 of 60 historical\n\nruns but answered `'Mercury'` cleanly when I called it today, so at least some\n\nof this moves over time and is not a fixed property of the model.\n\nThere is also a real question about what the fix should be, and I do not think\n\nit is obviously \"loosen the grader\". A grader that accepts `'Mercury.'` also\n\naccepts a model that ignores the instruction, and the instruction was part of\n\nthe task. Making the grader lenient would convert a visible measurement problem\n\ninto an invisible one. The alternative is to say plainly that these are\n\ninstruction-following tasks and retag them, which changes what the board has\n\nbeen reporting for three months.\n\nIf you maintain an eval suite with exact-match graders on short answers: check\n\nwhat your passing models actually return, not just whether they passed. The\n\nboard I built recorded *which* tasks failed for three months and never once\n\nrecorded *what the model said*, which is why this took a live call to see. That\n\ngap is the more embarrassing half of this post.\n\nI build deterministic evaluation and verification tooling for LLM systems, and I am looking for my first full-time role in AI evaluation or QA engineering. Remote US Eastern, or Charleston SC. [https://erikhill.dev](https://erikhill.dev)\n\nLive board: [https://egnaro9.github.io/model-drift/](https://egnaro9.github.io/model-drift/)\n\nSuite, graders and runner: [https://github.com/egnaro9/model-drift](https://github.com/egnaro9/model-drift)\n\n*The finding was surfaced automatically by the drift detector. The investigation, the live calls and the judgement of whether it was worth saying are mine.*", "url": "https://wpnews.pro/news/my-factual-recall-tasks-were-scoring-format-not-facts", "canonical_source": "https://dev.to/agentdev9/my-factual-recall-tasks-were-scoring-format-not-facts-j4m", "published_at": "2026-09-23 23:15:35+00:00", "updated_at": "2026-09-23 23:28:31.560957+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-tools"], "entities": ["Claude Sonnet 5", "GPT-4o mini", "Llama 3.1 8B", "Claude Opus 4.8", "Grok 4.5"], "alternates": {"html": "https://wpnews.pro/news/my-factual-recall-tasks-were-scoring-format-not-facts", "markdown": "https://wpnews.pro/news/my-factual-recall-tasks-were-scoring-format-not-facts.md", "text": "https://wpnews.pro/news/my-factual-recall-tasks-were-scoring-format-not-facts.txt", "jsonld": "https://wpnews.pro/news/my-factual-recall-tasks-were-scoring-format-not-facts.jsonld"}}