cd /news/large-language-models/we-ran-a-5-model-24-question-study-f… · home › topics › large-language-models › article
[ARTICLE · art-172176] src=dev.to ↗ pub= topic=large-language-models verified=true sentiment=↑ positive

We ran a 5-model, 24-question study for $0.87 — and published the receipt

A developer behind MultiO ran a 24-question, five-model comparison (GPT, Claude, Gemini, Grok and Llama) with a Council judging pass and a rerun control, metering the whole study at $0.87 (866,245 micro-USD) on the platform's own per-message receipts. The study found 52 material divergences, with the models materially agreeing on only 8 of 24 questions, and disclosed that 10 panel calls were exact-cache replays charged at zero and that judging required a metered structured-extraction follow-up in 17 of 24 runs. The team argues the answers are cheap and the real cost lies in the structured read that surfaces where models split.

recommendation

Budget for the structured judging pass, not the queries: in this study 17 of 24 judge runs needed a metered extraction retry, so add format-retry handling and cost tracking to any multi-model comparison.

more in playbook: Evals and testing for LLM apps
by read2 min views8 publishedNov 18, 2026

Most multi-model comparisons never tell you what they cost to run. Ours did, because in MultiO every message carries a receipt — so the study's own bill is part of its method, and we published it.

The programme: the same 24 questions, unchanged, sent to five mid-tier models — GPT, Claude, Gemini, Grok and Llama — then a Council read per question to flag where the answers materially split, plus a rerun control (two questions, three fresh samples each through one model). The total metered cost was $0.87 — 866,245 micro-USD on the study account's own receipts.

The breakdown is the interesting part:

Two honesty details sit inside those numbers. Ten of the panel calls (the first two questions in the completed run) were exact-cache replays of the lost first attempt's answers — same models, same questions, generated minutes earlier — and were charged 0; the Council judging on them ran fresh. And the judging itself needed a metered extraction follow-up in 17 of 24 runs when the first pass missed the structured format. Neither fact changes the findings — 52 material divergences, models materially agreeing on only 8 of 24 questions — but both are in the published method because the receipt is the point: if a tool meters usage, the study run on it should be auditable down to the micro.

That is also the practical takeaway for anyone cross-checking models on a budget: the answers are not the expensive part. Asking five models a question is cheap; the cost is in the structured read that tells you where they split — and that read is exactly the part a single-model workflow skips entirely, at $0 and at the price of never seeing the disagreement.

Full study, method, cost breakdown and all caveats (including that one judge was also a panel member, the rotating judging sets, and a memory-bleed incident we found and disclosed): https://multio.ai/research/model-divergence

Disclosure: I build MultiO (multio.ai) — One subscription. Every top AI model… Auto sends each message to the most cost-efficient model that can handle it. Compare asks up to three models side by side, and every message shows its exact token charge on a receipt. Free trial: 3,000,000 tokens, no card required.

── more in #large-language-models 4 stories · sorted by recency
── more on @multio 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/we-ran-a-5-model-24-…] indexed:0 read:2min 2026-11-18 · —