A common sanity check for AI answers is to ask again — regenerate, rephrase, run it a second time and see if the answer holds. Our divergence study included a control that suggests that check cannot catch the failure mode that matters most.
We sent the same 24 questions, unchanged, to five models — GPT, Claude, Gemini, Grok and Llama. Across the 24, there were 52 material divergences, and exactly half of them (26) involved a lone model: 14 claims exactly one model made, and 12 stances exactly one model took against the rest.
Then the control: we re-ran two questions — one factual, one recommendation — three times each through a single model, in identical form, as genuine fresh samples (the requests were cache-ineligible and every sample came back cacheHit: false). The model never changed its verdict. The run-to-run variance was wording, not substance.
That is the structural problem. The disagreement in this study lives between models, not inside one model's sampling — so asking the same model twice, or ten times, replays the same lone claim with the same confidence and never flags that it is alone.
Two lone elements from the study that a re-ask could never surface:
The hand-check is why this matters in both directions: of the 14 sole-source claims, 8 were the highest-value line in their set and 3 were wrong or invented. The lone voice is sometimes the only falsifiable condition in the room and sometimes an internal contradiction — and neither a re-ask nor a synthesis that averages the panel away will tell you which. A synthesis deletes the outlier claim, and the outlier was the best line 57% of the time.
My practical rule now: re-asking the same model checks its consistency, nothing more. For a decision that costs money to get wrong, the check that actually adds information is a different model — and a record of where they split.
Full study, method and caveats (including that one judge was also a panel member, and a memory-bleed incident we found and disclosed): https://multio.ai/research/model-divergence
Disclosure: I build MultiO (multio.ai) — One subscription. Every top AI model… Auto sends each message to the most cost-efficient model that can handle it. Compare asks up to three models side by side, and every message shows its exact token charge on a receipt. Free trial: 3,000,000 tokens, no card required.