{"slug": "we-ran-a-5-model-24-question-study-for-0-87-and-published-the-receipt", "title": "We ran a 5-model, 24-question study for $0.87 — and published the receipt", "summary": "A developer behind MultiO ran a 24-question, five-model comparison (GPT, Claude, Gemini, Grok and Llama) with a Council judging pass and a rerun control, metering the whole study at $0.87 (866,245 micro-USD) on the platform's own per-message receipts. The study found 52 material divergences, with the models materially agreeing on only 8 of 24 questions, and disclosed that 10 panel calls were exact-cache replays charged at zero and that judging required a metered structured-extraction follow-up in 17 of 24 runs. The team argues the answers are cheap and the real cost lies in the structured read that surfaces where models split.", "body_md": "Most multi-model comparisons never tell you what they cost to run. Ours did, because in MultiO every message carries a receipt — so the study's own bill is part of its method, and we published it.\n\nThe programme: the same 24 questions, unchanged, sent to five mid-tier models — GPT, Claude, Gemini, Grok and Llama — then a Council read per question to flag where the answers materially split, plus a rerun control (two questions, three fresh samples each through one model). The total metered cost was $0.87 — 866,245 micro-USD on the study account's own receipts.\n\nThe breakdown is the interesting part:\n\nTwo honesty details sit inside those numbers. Ten of the panel calls (the first two questions in the completed run) were exact-cache replays of the lost first attempt's answers — same models, same questions, generated minutes earlier — and were charged 0; the Council judging on them ran fresh. And the judging itself needed a metered extraction follow-up in 17 of 24 runs when the first pass missed the structured format. Neither fact changes the findings — 52 material divergences, models materially agreeing on only 8 of 24 questions — but both are in the published method because the receipt is the point: if a tool meters usage, the study run on it should be auditable down to the micro.\n\nThat is also the practical takeaway for anyone cross-checking models on a budget: the answers are not the expensive part. Asking five models a question is cheap; the cost is in the structured read that tells you where they split — and that read is exactly the part a single-model workflow skips entirely, at $0 and at the price of never seeing the disagreement.\n\nFull study, method, cost breakdown and all caveats (including that one judge was also a panel member, the rotating judging sets, and a memory-bleed incident we found and disclosed): [https://multio.ai/research/model-divergence](https://multio.ai/research/model-divergence)\n\nDisclosure: I build MultiO (multio.ai) — One subscription. Every top AI model… Auto sends each message to the most cost-efficient model that can handle it. Compare asks up to three models side by side, and every message shows its exact token charge on a receipt. Free trial: 3,000,000 tokens, no card required.", "url": "https://wpnews.pro/news/we-ran-a-5-model-24-question-study-for-0-87-and-published-the-receipt", "canonical_source": "https://dev.to/getmultio/we-ran-a-5-model-24-question-study-for-087-and-published-the-receipt-1ibo", "published_at": "2026-11-18 22:59:36+00:00", "updated_at": "2026-11-18 23:11:47.845572+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-products"], "entities": ["MultiO", "GPT", "Claude", "Gemini", "Grok", "Llama"], "also_reported_by": [], "takeaway": "Running the same 24 questions through five mid-tier models cost $0.87, but the structured Council judging needed a metered extraction follow-up in 17 of 24 runs, so budget for the disagreement analysis rather than the queries.", "recommendation": {"advice": "Budget for the structured judging pass, not the queries: in this study 17 of 24 judge runs needed a metered extraction retry, so add format-retry handling and cost tracking to any multi-model comparison.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/we-ran-a-5-model-24-question-study-for-0-87-and-published-the-receipt", "markdown": "https://wpnews.pro/news/we-ran-a-5-model-24-question-study-for-0-87-and-published-the-receipt.md", "text": "https://wpnews.pro/news/we-ran-a-5-model-24-question-study-for-0-87-and-published-the-receipt.txt", "jsonld": "https://wpnews.pro/news/we-ran-a-5-model-24-question-study-for-0-87-and-published-the-receipt.jsonld"}}