Every week, the largest AI models are asked the question:
One word answer.System prompt
- Claude Opus 5AnthropicNo.reasoning2.8 sReasoned for 2.8 s (100% of the call) on 118 tokens, then answered · 3 of 3 runs agreed
- Claude Sonnet 5AnthropicYes.reasoning1.3 sReasoned for 1.3 s (99% of the call) on 20 tokens, then answered · 2 of 3 runs agreed
- Claude Haiku 4.5AnthropicYes.reasoning674 msNo reasoning, answered straight away · 3 of 3 runs agreed
- GPT-5.6 SolOpenAIYes.reasoning1.5 sReasoned for 1.2 s (84% of the call) on 16 tokens, then answered · 3 of 3 runs agreed
- GPT-5.5OpenAIYesreasoning1.4 sReasoned for 1.2 s (86% of the call) on 33 tokens, then answered · 3 of 3 runs agreed
- GPT-5.4 miniOpenAIYesreasoning1.6 sNo reasoning, answered straight away · 2 of 3 runs agreed
- Grok 4.6xAIYesreasoning8.7 sReasoned for 8.7 s (100% of the call) on 365 tokens, then answered · 2 of 3 runs agreed
- Grok 4.3xAINoreasoning8.6 sReasoned for 8.6 s (100% of the call) on 574 tokens, then answered · 2 of 3 runs agreed
- Grok 4.20 (non-reasoning)xAIYes.reasoning407 msNo reasoning, answered straight away · 2 of 3 runs agreed
- Mistral Medium 3.5Mistral AINo.reasoning368 msNo reasoning, answered straight away · 3 of 3 runs agreed
- Mistral Small 4Mistral AINoreasoning358 msNo reasoning, answered straight away · 2 of 3 runs agreed
- DeepSeek V4 ProDeepSeekNo.reasoning2.2 sReasoned for 2.2 s (98% of the call) on 149 tokens, then answered · 3 of 3 runs agreed
7 said yes
5 said no
Recorded week 36, 2026. Real durations, verbatim words. Teal is the wait before the first word, hatched where the model spent it reasoning; the rest is answering.Read the report →
Same question, different minds
They do not agree with each other. #
| Question | Claude Opus 5 | Claude Sonnet 5 | Claude Haiku 4.5 | GPT-5.6 Sol | GPT-5.5 | GPT-5.4 mini | Grok 4.6 | Grok 4.3 | Grok 4.20 (non-reasoning) | Mistral Medium 3.5 | Mistral Small 4 | DeepSeek V4 Pro | Agree |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
NoYesYesYesYesYesYesNoYesNoNoNohamburger
YesYesYesYesYesYesYesYesYesYesYesYestaco
NoNoNoNoNoNoNoNoYesNoNoNogrilled cheese
YesYesYesYesYesYesYesYesYesYesYesYeswrap
NoYesYesYesYesYesNoNoYesNoNoNotuna melt
YesYesYesYesYesYesYesYesYesYesYesYesRead the 6 reports →One straight-faced analyst report per question: standings, the certainty quadrant, every verbatim answer under every framing, and a PDF for each.
Tell them the answer
Some of them believe you. #
Share of questions where a model changed its answer once a system prompt stated the answer as fact. Holding firm and following instructions are both defensible; the methodology grades neither.
- GPT-5.6 Sol50%6 of 12
- GPT-5.550%6 of 12
- GPT-5.4 mini50%6 of 12
- Mistral Medium 3.550%6 of 12
- Mistral Small 450%6 of 12
- Claude Sonnet 533%4 of 12
- Grok 4.20 (non-reasoning)33%3 of 9
- Claude Haiku 4.525%3 of 12
- Claude Opus 517%2 of 12
- Grok 4.617%2 of 12
- Grok 4.317%2 of 12
- DeepSeek V4 Pro17%2 of 12
Submit your own question
Ask the models something. #
Send it in. An accepted question appears here under Up next, credited to you if you want, then joins an edition and gets its own report. Every question is asked the same way, so it ends with One word answer.
; we add that if you leave it off.
Where it goes:
Open it as a GitHub issuethe question goes into the form, ready to fileSend it to En Dash Consultinga contact form with your question in it; leave an email address to hear when it goes liveSuggest a model insteadthe add-a-model form asks for what the registry needs
Open source
Point it at your own question. #
One repo, MIT-licensed: adapters for every provider, the framings, the site. Clone it, swap the question, add whatever keys you have, and you get the same cross-model, cross-framing analysis for cents. Pull requests welcome.
GitHubSelf-hostingAdd a modelContributing
Have a question the models should get? Send it in.
git clone https://github.com/en-dash-consulting/hotdogbenchmark.git
cd hotdogbenchmark && npm install
npm run bench -- run --mock --out tmp/mock-run.json
npm run dev
Week 36, 2026 · published September 3, 2026 · one edition so far