# We ran a 5-model, 24-question study for $0.87 — and published the receipt

> Source: <https://dev.to/getmultio/we-ran-a-5-model-24-question-study-for-087-and-published-the-receipt-1ibo>
> Published: 2026-11-18 22:59:36+00:00

**Recommendation:** Budget for the structured judging pass, not the queries: in this study 17 of 24 judge runs needed a metered extraction retry, so add format-retry handling and cost tracking to any multi-model comparison.
More in the playbook "Evals and testing for LLM apps": <https://wpnews.pro/playbooks/evals-testing>

Most multi-model comparisons never tell you what they cost to run. Ours did, because in MultiO every message carries a receipt — so the study's own bill is part of its method, and we published it.

The programme: the same 24 questions, unchanged, sent to five mid-tier models — GPT, Claude, Gemini, Grok and Llama — then a Council read per question to flag where the answers materially split, plus a rerun control (two questions, three fresh samples each through one model). The total metered cost was $0.87 — 866,245 micro-USD on the study account's own receipts.

The breakdown is the interesting part:

Two honesty details sit inside those numbers. Ten of the panel calls (the first two questions in the completed run) were exact-cache replays of the lost first attempt's answers — same models, same questions, generated minutes earlier — and were charged 0; the Council judging on them ran fresh. And the judging itself needed a metered extraction follow-up in 17 of 24 runs when the first pass missed the structured format. Neither fact changes the findings — 52 material divergences, models materially agreeing on only 8 of 24 questions — but both are in the published method because the receipt is the point: if a tool meters usage, the study run on it should be auditable down to the micro.

That is also the practical takeaway for anyone cross-checking models on a budget: the answers are not the expensive part. Asking five models a question is cheap; the cost is in the structured read that tells you where they split — and that read is exactly the part a single-model workflow skips entirely, at $0 and at the price of never seeing the disagreement.

Full study, method, cost breakdown and all caveats (including that one judge was also a panel member, the rotating judging sets, and a memory-bleed incident we found and disclosed): [https://multio.ai/research/model-divergence](https://multio.ai/research/model-divergence)

Disclosure: I build MultiO (multio.ai) — One subscription. Every top AI model… Auto sends each message to the most cost-efficient model that can handle it. Compare asks up to three models side by side, and every message shows its exact token charge on a receipt. Free trial: 3,000,000 tokens, no card required.
