Show HN: Hot. Dog. Bench. Mark. The AI benchmark we deserve

A new benchmark asking 12 leading AI models whether a hot dog is a sandwich found that 7 said yes and 5 said no, with results varying by model and reasoning time. The benchmark, recorded in week 36 of 2026, tested models from Anthropic, OpenAI, xAI, Mistral AI, and DeepSeek, with responses ranging from Claude Opus 5's 'No' after 2.8 seconds of reasoning to Grok 4.20's 'Yes' in 407 milliseconds.

Every week, the largest AI models are asked the question: Is a hot dog a sandwich? One word answer.System prompt - Claude Opus 5AnthropicNo.reasoning2.8 sReasoned for 2.8 s 100% of the call on 118 tokens, then answered · 3 of 3 runs agreed - Claude Sonnet 5AnthropicYes.reasoning1.3 sReasoned for 1.3 s 99% of the call on 20 tokens, then answered · 2 of 3 runs agreed - Claude Haiku 4.5AnthropicYes.reasoning674 msNo reasoning, answered straight away · 3 of 3 runs agreed - GPT-5.6 SolOpenAIYes.reasoning1.5 sReasoned for 1.2 s 84% of the call on 16 tokens, then answered · 3 of 3 runs agreed - GPT-5.5OpenAIYesreasoning1.4 sReasoned for 1.2 s 86% of the call on 33 tokens, then answered · 3 of 3 runs agreed - GPT-5.4 miniOpenAIYesreasoning1.6 sNo reasoning, answered straight away · 2 of 3 runs agreed - Grok 4.6xAIYesreasoning8.7 sReasoned for 8.7 s 100% of the call on 365 tokens, then answered · 2 of 3 runs agreed - Grok 4.3xAINoreasoning8.6 sReasoned for 8.6 s 100% of the call on 574 tokens, then answered · 2 of 3 runs agreed - Grok 4.20 non-reasoning xAIYes.reasoning407 msNo reasoning, answered straight away · 2 of 3 runs agreed - Mistral Medium 3.5Mistral AINo.reasoning368 msNo reasoning, answered straight away · 3 of 3 runs agreed - Mistral Small 4Mistral AINoreasoning358 msNo reasoning, answered straight away · 2 of 3 runs agreed - DeepSeek V4 ProDeepSeekNo.reasoning2.2 sReasoned for 2.2 s 98% of the call on 149 tokens, then answered · 3 of 3 runs agreed 7 said yes 5 said no Recorded week 36, 2026. Real durations, verbatim words. Teal is the wait before the first word, hatched where the model spent it reasoning; the rest is answering. Read the report → /reports/hot-dog/ Same question, different minds They do not agree with each other. | Question | Claude Opus 5 | Claude Sonnet 5 | Claude Haiku 4.5 | GPT-5.6 Sol | GPT-5.5 | GPT-5.4 mini | Grok 4.6 | Grok 4.3 | Grok 4.20 non-reasoning | Mistral Medium 3.5 | Mistral Small 4 | DeepSeek V4 Pro | Agree | |---|---|---|---|---|---|---|---|---|---|---|---|---|---| | No /reports/hot-dog/ profile-anthropic-claude-opus-5 Yes /reports/hot-dog/ profile-anthropic-claude-sonnet-5 Yes /reports/hot-dog/ profile-anthropic-claude-haiku-4-5-20251001 Yes /reports/hot-dog/ profile-openai-gpt-5-6-sol Yes /reports/hot-dog/ profile-openai-gpt-5-5 Yes /reports/hot-dog/ profile-openai-gpt-5-4-mini Yes /reports/hot-dog/ profile-xai-grok-4-6 No /reports/hot-dog/ profile-xai-grok-4-3 Yes /reports/hot-dog/ profile-xai-grok-4-20-0309-non-reasoning No /reports/hot-dog/ profile-mistral-mistral-medium-2604 No /reports/hot-dog/ profile-mistral-mistral-small-2603 No /reports/hot-dog/ profile-deepseek-deepseek-v4-pro hamburger /reports/hamburger/ Yes /reports/hamburger/ profile-anthropic-claude-opus-5 Yes /reports/hamburger/ profile-anthropic-claude-sonnet-5 Yes /reports/hamburger/ profile-anthropic-claude-haiku-4-5-20251001 Yes /reports/hamburger/ profile-openai-gpt-5-6-sol Yes /reports/hamburger/ profile-openai-gpt-5-5 Yes /reports/hamburger/ profile-openai-gpt-5-4-mini Yes /reports/hamburger/ profile-xai-grok-4-6 Yes /reports/hamburger/ profile-xai-grok-4-3 Yes /reports/hamburger/ profile-xai-grok-4-20-0309-non-reasoning Yes /reports/hamburger/ profile-mistral-mistral-medium-2604 Yes /reports/hamburger/ profile-mistral-mistral-small-2603 Yes /reports/hamburger/ profile-deepseek-deepseek-v4-pro taco /reports/taco/ No /reports/taco/ profile-anthropic-claude-opus-5 No /reports/taco/ profile-anthropic-claude-sonnet-5 No /reports/taco/ profile-anthropic-claude-haiku-4-5-20251001 No /reports/taco/ profile-openai-gpt-5-6-sol No /reports/taco/ profile-openai-gpt-5-5 No /reports/taco/ profile-openai-gpt-5-4-mini No /reports/taco/ profile-xai-grok-4-6 No /reports/taco/ profile-xai-grok-4-3 Yes /reports/taco/ profile-xai-grok-4-20-0309-non-reasoning No /reports/taco/ profile-mistral-mistral-medium-2604 No /reports/taco/ profile-mistral-mistral-small-2603 No /reports/taco/ profile-deepseek-deepseek-v4-pro grilled cheese /reports/grilled-cheese/ Yes /reports/grilled-cheese/ profile-anthropic-claude-opus-5 Yes /reports/grilled-cheese/ profile-anthropic-claude-sonnet-5 Yes /reports/grilled-cheese/ profile-anthropic-claude-haiku-4-5-20251001 Yes /reports/grilled-cheese/ profile-openai-gpt-5-6-sol Yes /reports/grilled-cheese/ profile-openai-gpt-5-5 Yes /reports/grilled-cheese/ profile-openai-gpt-5-4-mini Yes /reports/grilled-cheese/ profile-xai-grok-4-6 Yes /reports/grilled-cheese/ profile-xai-grok-4-3 Yes /reports/grilled-cheese/ profile-xai-grok-4-20-0309-non-reasoning Yes /reports/grilled-cheese/ profile-mistral-mistral-medium-2604 Yes /reports/grilled-cheese/ profile-mistral-mistral-small-2603 Yes /reports/grilled-cheese/ profile-deepseek-deepseek-v4-pro wrap /reports/wrap/ No /reports/wrap/ profile-anthropic-claude-opus-5 Yes /reports/wrap/ profile-anthropic-claude-sonnet-5 Yes /reports/wrap/ profile-anthropic-claude-haiku-4-5-20251001 Yes /reports/wrap/ profile-openai-gpt-5-6-sol Yes /reports/wrap/ profile-openai-gpt-5-5 Yes /reports/wrap/ profile-openai-gpt-5-4-mini No /reports/wrap/ profile-xai-grok-4-6 No /reports/wrap/ profile-xai-grok-4-3 Yes /reports/wrap/ profile-xai-grok-4-20-0309-non-reasoning No /reports/wrap/ profile-mistral-mistral-medium-2604 No /reports/wrap/ profile-mistral-mistral-small-2603 No /reports/wrap/ profile-deepseek-deepseek-v4-pro tuna melt /reports/tuna-melt/ Yes /reports/tuna-melt/ profile-anthropic-claude-opus-5 Yes /reports/tuna-melt/ profile-anthropic-claude-sonnet-5 Yes /reports/tuna-melt/ profile-anthropic-claude-haiku-4-5-20251001 Yes /reports/tuna-melt/ profile-openai-gpt-5-6-sol Yes /reports/tuna-melt/ profile-openai-gpt-5-5 Yes /reports/tuna-melt/ profile-openai-gpt-5-4-mini Yes /reports/tuna-melt/ profile-xai-grok-4-6 Yes /reports/tuna-melt/ profile-xai-grok-4-3 Yes /reports/tuna-melt/ profile-xai-grok-4-20-0309-non-reasoning Yes /reports/tuna-melt/ profile-mistral-mistral-medium-2604 Yes /reports/tuna-melt/ profile-mistral-mistral-small-2603 Yes /reports/tuna-melt/ profile-deepseek-deepseek-v4-pro Read the 6 reports → /reports/ One straight-faced analyst report per question: standings, the certainty quadrant, every verbatim answer under every framing, and a PDF for each. Tell them the answer Some of them believe you. Share of questions where a model changed its answer once a system prompt stated the answer as fact. Holding firm and following instructions are both defensible; the methodology /methodology/ sensitivity grades neither. - GPT-5.6 Sol50%6 of 12 - GPT-5.550%6 of 12 - GPT-5.4 mini50%6 of 12 - Mistral Medium 3.550%6 of 12 - Mistral Small 450%6 of 12 - Claude Sonnet 533%4 of 12 - Grok 4.20 non-reasoning 33%3 of 9 - Claude Haiku 4.525%3 of 12 - Claude Opus 517%2 of 12 - Grok 4.617%2 of 12 - Grok 4.317%2 of 12 - DeepSeek V4 Pro17%2 of 12 Submit your own question Ask the models something. Send it in. An accepted question appears here under Up next , credited to you if you want, then joins an edition and gets its own report. Every question is asked the same way, so it ends with One word answer. ; we add that if you leave it off. Where it goes: Open it as a GitHub issue https://github.com/en-dash-consulting/hotdogbenchmark/issues/new?template=add question.yml the question goes into the form, ready to file Send it to En Dash Consulting https://endash.us/?showContact=true&contactSource=hotdogbenchmark-lol&contactTitle=Submit+a+question+to+the+Hotdog+Benchmark&contactMessage=Question+for+the+Hotdog+Benchmark%3A%0A%28your+question+here%29%0A%0ASubject%2C+as+it+reads+in+a+sentence%3A+%28e.g.+%22a+burrito%22%29%0AWhy+it+is+worth+asking%3A%0ACredit+me+as+%28or+say+%22no+credit%22%29%3A%0AEmail+me+when+it+goes+live+at%3A a contact form with your question in it; leave an email address to hear when it goes live Suggest a model instead https://github.com/en-dash-consulting/hotdogbenchmark/issues/new?template=add model or provider.yml the add-a-model form asks for what the registry needs Open source Point it at your own question. One repo, MIT-licensed: adapters for every provider, the framings, the site. Clone it, swap the question, add whatever keys you have, and you get the same cross-model, cross-framing analysis for cents. Pull requests welcome. GitHub https://github.com/en-dash-consulting/hotdogbenchmark Self-hosting https://github.com/en-dash-consulting/hotdogbenchmark/blob/main/docs/self-hosting.md Add a model /add-a-model/ Contributing https://github.com/en-dash-consulting/hotdogbenchmark/blob/main/CONTRIBUTING.md Have a question the models should get? Send it in ask . git clone https://github.com/en-dash-consulting/hotdogbenchmark.git cd hotdogbenchmark && npm install npm run bench -- run --mock --out tmp/mock-run.json npm run dev Week 36, 2026 · published September 3, 2026 · one edition so far /runs/