Originally published at erikhill.dev. The numbers below are checked against the repository they come from.
This is a finding about my own harness. The suspect is the probe, not the models it measures.
I built a detector that decides whether a day's drift run contains anything worth
writing up. The first thing it did was accuse my own suite. One task,
fact-element, had failed on three or more providers on the same day, on 46
separate days.
The task is this:
What is the chemical symbol for gold? Two letters only.
Graded by exact match against au, after stripping whitespace and lowercasing.
Claude Sonnet 5 failed it 60 times out of 60. GPT-4o mini failed it 59 out of 59. Llama 3.1 8B, 23 out of 23. Those are not knowledge results. A model
| claim | how it was checked |
|---|---|
fact-element failed on 3 or more providers on the same day, 46 times |
cross-provider flip analysis over every stored run in dashboard/drift_board.json |
| those were graded failures, not outages | every run counted had reliability: 1.0 , so no call errored and none was truncated |
| the models know the answer | gpt-4o-mini answered'The chemical symbol for gold is Au.' ;claude-sonnet-5 answered'**Au**\n\nThat's the chemical symbol for gold, derived from the Latin word *aurum*.' |
| the same models pass a near-identical task | all three answered fact-capital with exactly'Tokyo' , 0 failures in 59 and 60 runs |
| one trailing period is the whole difference | gpt-4o-mini answeredfact-planet with'Mercury.' and was recorded wrong 59 times out of 59 |
| a model that formats tersely passes cleanly | claude-opus-4-8 answered'Au' ,'Mercury' ,'Tokyo' , and failsfact-element on 1 of 60 runs |
The answers above are not from the board. They came from nine live calls made
through the probe's own call path, printed with repr() so punctuation and
markdown are visible rather than inferred.
fact-capital is the control, and it is the reason this is a finding rather
than a theory. It asks for the capital of Japan and every model in the table
answers 'Tokyo', exactly, on every run. Same grader, same strictness, same
models, same day. Nothing about exact matching is broken in general.
The three tasks differ in one way. fact-capital says "Answer with only the city name", which is an instruction about the shape of the output.
fact-element says fact-planet says The suite has 35 tasks, so the smallest accuracy change it can print is
100/35 = 2.86 points. fact-element and fact-planet together are 5.7 points
of every affected model's score, permanently, for reasons that have nothing to
do with the models.
Worse than the points: these tasks are tagged factual-recall, and that tag
feeds a per-category breakdown on the public board. GPT-4o mini's
factual-recall score has been reading as a knowledge number when two thirds of
it is a formatting number. Anyone comparing models on that column, including
me, was comparing how terse they are.
The part I am least sure of is the explanation, not the measurement. The numbers
are solid and the live answers are quoted verbatim, but "models treat answer with only X as a constraint and
Two things genuinely argue against my reading. Grok 4.5 fails fact-element on
26 of 49 runs, which is close to a coin flip, and a rigid house style should not
produce that. And Claude Sonnet 5 failed fact-planet on 38 of 60 historical
runs but answered 'Mercury' cleanly when I called it today, so at least some
of this moves over time and is not a fixed property of the model.
There is also a real question about what the fix should be, and I do not think
it is obviously "loosen the grader". A grader that accepts 'Mercury.' also
accepts a model that ignores the instruction, and the instruction was part of
the task. Making the grader lenient would convert a visible measurement problem
into an invisible one. The alternative is to say plainly that these are
instruction-following tasks and retag them, which changes what the board has
been reporting for three months.
If you maintain an eval suite with exact-match graders on short answers: check what your passing models actually return, not just whether they passed. The
board I built recorded which tasks failed for three months and never once
recorded what the model said, which is why this took a live call to see. That
gap is the more embarrassing half of this post.
I build deterministic evaluation and verification tooling for LLM systems, and I am looking for my first full-time role in AI evaluation or QA engineering. Remote US Eastern, or Charleston SC. https://erikhill.dev
Live board: [https://egnaro9.github.io/model-drift/](https://egnaro9.github.io/model-drift/)
Suite, graders and runner: [https://github.com/egnaro9/model-drift](https://github.com/egnaro9/model-drift)
The finding was surfaced automatically by the drift detector. The investigation, the live calls and the judgement of whether it was worth saying are mine.