A model release announcement is a marketing document containing real information. Both halves of that sentence are true, and reading one well is mostly a matter of knowing which sentences are which.
Sort every sentence in the post into one of these before reacting to any of it. The categories differ enormously in how much they should move you.
| Kind of claim | Description |
|---|---|
| Structural fact | Context window, modalities supported, whether it does tool calling, whether reasoning is exposed, knowledge cutoff, price, availability. Checkable, stable, and the most useful part of the document. Read this first. |
| Benchmark number | A score on a named evaluation. Real but narrow: it tells you the model does well on that evaluation, under conditions usually only partly described. Almost never transfers directly to your task. |
| Qualitative capability claim | 'Better at following complex instructions', 'improved coding ability', 'more reliable agentic behaviour'. Directionally informative, unfalsifiable as stated. Treat as a hypothesis to test, not a finding. |
| Comparative positioning | Claims against other vendors' models, chosen and configured by the party making the claim. The comparison set is the message: notice which competitors and which of their models are absent. |
The practical consequence: a release post is at its most useful in its most boring paragraphs. Price, context limit and modality decide whether the model is even a candidate for your system, and they are facts. Everything above them on the page is a claim about quality that you will have to test anyway.
Benchmarks are not worthless — they catch gross incapability and they track progress across generations. They are just being asked to carry far more weight than they can. Five questions to ask of any table:
Releases are consistent in what they omit, which makes the omissions readable. Note each one, because each corresponds to a question you will have to answer yourself before adopting anything.
The whole point of reading critically is to end with an action, and the action is almost always the same: run your own evaluation. A release post can tell you whether it is worth the afternoon; it cannot tell you the answer.
The procedure, which is cheap if you have a frozen set already and is the best argument for having one:
1. Check the structural facts first. If the context limit,
the price or the modality rules it out, stop here.
2. Run your existing frozen set against the new model with
your CURRENT prompt. This is the honest baseline: it is
what switching with no other work would give you.
3. Run it again with the prompt lightly adapted, if the
release describes a different prompting convention.
Record both numbers; the difference between them is a
real cost of switching.
4. Measure cost per request and latency at the same time,
from the same run. You already have the requests.
5. Decide against the three axes together: quality, cost,
latency. A model that wins on one and loses on another
is the normal case, not an anomaly.
Step 2 is the one people skip, and it is the one that produces the surprising results in both directions — sometimes a heavily promoted release is worse on your task with your prompt, and sometimes an unremarkable one is much better because your task happens to sit where it improved.
Switching is not free even when the new model is better. The costs are a re-evaluation, possible prompt adaptation, a period of unfamiliarity with the new failure modes, and the risk that the new model is itself updated soon. So the bar is higher than “it scored more”.
And whichever you decide, pin the version explicitly rather than following a moving alias. An alias that silently resolves to a newer model is a deploy you did not make, and it is the way most teams discover that their evaluation was the only thing standing between them and a quiet regression.
The structural facts in step 1 are the part of a release that is checkable, and they are easier to check side by side than in a sequence of announcement posts — a catalogue listing context limits, modalities and per-token prices per model answers the “is this even a candidate” question in about a minute, which is the right amount of time to spend on it.