Member-only story
DeepSeek’s own chart says 82.7 on Terminal-Bench 2.1. Artificial Analysis measured 79. That 3.7-point gap is 2.2x the margin the 82.7 was defending.
On July 31, 2026, DeepSeek shipped DeepSeek-V4-Flash-0731
and published a nine-row agent benchmark chart. The chart is genuinely impressive: a 284-billion-parameter model with 13 billion active at inference beats Z.ai's GLM-5.2 on every single row where both were measured, and comes within a few points of Claude Opus 4.8 on several.
Then I read the footnote under the table.
For public Code Agent tasks, V4-Flash was tested using our upcoming DeepSeek Harness (minimal mode) framework. Settings: max tier, topp=0.95, temperature=1.0. Upcoming. The framework that produced every number on that chart had not shipped on the day the chart shipped. No one outside DeepSeek could run it.
That would be a footnote-sized problem if agent benchmark scores were harness-independent. They are not. The margin DeepSeek is claiming on Terminal-Bench 2.1 is 1.7 points (82.7 vs GLM-5.2’s 81.0). The smallest harness-only effect I could find documented anywhere in the 2026 literature — adding one search subagent to an otherwise identical scaffold — is 2.1 points. The floor of the…