# My factual-recall tasks were scoring format, not facts

> Source: <https://dev.to/agentdev9/my-factual-recall-tasks-were-scoring-format-not-facts-j4m>
> Published: 2026-09-23 23:15:35+00:00

*Originally published at [erikhill.dev](https://erikhill.dev/notes/2026-09-22-my-factual-recall-tasks-were-scoring-format-not-facts/). The numbers below are checked against the repository they come from.*

**This is a finding about my own harness. The suspect is the probe, not the models it measures.**

I built a detector that decides whether a day's drift run contains anything worth

writing up. The first thing it did was accuse my own suite. One task,

`fact-element`, had failed on three or more providers on the same day, on 46

separate days.

The task is this:

What is the chemical symbol for gold? Two letters only.

Graded by exact match against `au`, after stripping whitespace and lowercasing.

Claude Sonnet 5 failed it **60 times out of 60**. GPT-4o mini failed it **59 out of 59**. Llama 3.1 8B, 23 out of 23. Those are not knowledge results. A model

| claim | how it was checked | 
|---|---|
| `fact-element` failed on 3 or more providers on the same day, 46 times | cross-provider flip analysis over every stored run in `dashboard/drift_board.json` | 
| those were graded failures, not outages | every run counted had `reliability: 1.0` , so no call errored and none was truncated | 
| the models know the answer | `gpt-4o-mini` answered`'The chemical symbol for gold is Au.'` ;`claude-sonnet-5` answered`'**Au**\n\nThat's the chemical symbol for gold, derived from the Latin word *aurum*.'` | 
| the same models pass a near-identical task | all three answered `fact-capital` with exactly`'Tokyo'` , 0 failures in 59 and 60 runs | 
| one trailing period is the whole difference | `gpt-4o-mini` answered`fact-planet` with`'Mercury.'` and was recorded wrong 59 times out of 59 | 
| a model that formats tersely passes cleanly | `claude-opus-4-8` answered`'Au'` ,`'Mercury'` ,`'Tokyo'` , and fails`fact-element` on 1 of 60 runs | 

The answers above are not from the board. They came from nine live calls made

through the probe's own call path, printed with `repr()` so punctuation and

markdown are visible rather than inferred.

`fact-capital` is the control, and it is the reason this is a finding rather

than a theory. It asks for the capital of Japan and every model in the table

answers `'Tokyo'`, exactly, on every run. Same grader, same strictness, same

models, same day. Nothing about exact matching is broken in general.

The three tasks differ in one way. `fact-capital` says **"Answer with only the city name"**, which is an instruction about the shape of the output.

`fact-element` says `fact-planet` says The suite has 35 tasks, so the smallest accuracy change it can print is

100/35 = 2.86 points. `fact-element` and `fact-planet` together are 5.7 points

of every affected model's score, permanently, for reasons that have nothing to

do with the models.

Worse than the points: these tasks are tagged `factual-recall`, and that tag

feeds a per-category breakdown on the public board. GPT-4o mini's

factual-recall score has been reading as a knowledge number when two thirds of

it is a formatting number. Anyone comparing models on that column, including

me, was comparing how terse they are.

The part I am least sure of is the explanation, not the measurement. The numbers

are solid and the live answers are quoted verbatim, but "models treat *answer with only X* as a constraint and 

Two things genuinely argue against my reading. Grok 4.5 fails `fact-element` on

26 of 49 runs, which is close to a coin flip, and a rigid house style should not

produce that. And Claude Sonnet 5 failed `fact-planet` on 38 of 60 historical

runs but answered `'Mercury'` cleanly when I called it today, so at least some

of this moves over time and is not a fixed property of the model.

There is also a real question about what the fix should be, and I do not think

it is obviously "loosen the grader". A grader that accepts `'Mercury.'` also

accepts a model that ignores the instruction, and the instruction was part of

the task. Making the grader lenient would convert a visible measurement problem

into an invisible one. The alternative is to say plainly that these are

instruction-following tasks and retag them, which changes what the board has

been reporting for three months.

If you maintain an eval suite with exact-match graders on short answers: check

what your passing models actually return, not just whether they passed. The

board I built recorded *which* tasks failed for three months and never once

recorded *what the model said*, which is why this took a live call to see. That

gap is the more embarrassing half of this post.

I build deterministic evaluation and verification tooling for LLM systems, and I am looking for my first full-time role in AI evaluation or QA engineering. Remote US Eastern, or Charleston SC. [https://erikhill.dev](https://erikhill.dev)

Live board: [https://egnaro9.github.io/model-drift/](https://egnaro9.github.io/model-drift/)

Suite, graders and runner: [https://github.com/egnaro9/model-drift](https://github.com/egnaro9/model-drift)

*The finding was surfaced automatically by the drift detector. The investigation, the live calls and the judgement of whether it was worth saying are mine.*