{"slug": "do-ai-assistants-give-the-same-answer-twice-we-asked-20-questions-three-times", "title": "Do AI assistants give the same answer twice? We asked 20 questions three times", "summary": "A benchmark called PlaceBench tested six AI assistants on 20 everyday questions about real places, asking each question three times, and found that answer consistency varied widely: Dots answered 17 of 20 correctly on all three tries, while Instinct managed only 5. The team reports pass^3 scores ranging from 85% for Dots to 25% for Instinct, and notes that assistants with persistent memory or continuous conversations can draw on earlier answers when repeating a question. The questions, answer keys, sources and grades are published under CC BY 4.0.", "body_md": "Ask an AI assistant the same question twice and you may get two different answers. In PlaceBench, our test of six assistants on everyday questions about real places, we asked 20 of the 150 questions three times each to see how often the answer holds.\n\n| Assistant | Right on the first try | Right all 3 times | Grade changed between tries | Wrong at least once | \n|---|---|---|---|---|\n| Dots | 18 of 20 | **17** | 3 | 0 | \n| Claude Duet | 17 | 13 | 6 | 4 | \n| Muse | 15 | 12 | 7 | 2 | \n| Ask Maps | 14 | 12 | 5 | 5 | \n| Instinct | 14 | 5 | 15 | 7 | \n| Grok Bot | 13 | 8 | 11 | 6 | \n\n\"Grade changed\" counts the questions where the three answers didn't all get the same grade. We computed the last two columns from the public grades file.\n\nThe methodology also reports pass^k, the chance that k tries at a question are all right. For three tries: Dots 85%, Claude Duet 65%, Muse 60%, Ask Maps 60%, Grok Bot 40%, Instinct 25%.\n\nIf you build on top of an assistant, one good answer in testing isn't enough. Ask the same question several times, or check the facts the answer depends on, such as whether the place is open at that hour.\n\n20 questions is a small sample. Instinct keeps one continuous conversation and Dots keeps memory between chats, so their repeats can draw on earlier answers. We fixed the 20 questions before collecting any answers.\n\nQuestions, answer keys, sources and every grade, under CC BY 4.0: [github.com/voygr-tech/placebench](https://github.com/voygr-tech/placebench). Full write-up: [voygr.tech/placebench](https://voygr.tech/placebench/).", "url": "https://wpnews.pro/news/do-ai-assistants-give-the-same-answer-twice-we-asked-20-questions-three-times", "canonical_source": "https://dev.to/voygr/do-ai-assistants-give-the-same-answer-twice-we-asked-20-questions-three-times-1j5n", "published_at": "2026-11-20 10:31:45+00:00", "updated_at": "2026-11-20 10:42:32.273675+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "ai-tools"], "entities": ["PlaceBench", "Dots", "Claude Duet", "Muse", "Ask Maps", "Instinct", "Grok Bot", "Voygr"], "also_reported_by": [], "takeaway": "Across 20 questions asked three times each, only Dots answered 17 of 20 correctly every time, while Instinct answered just 5 of 20 correctly on all three tries and changed grade on 15 questions.", "recommendation": {"advice": "Repeat each question several times in testing and verify the underlying facts, since a single correct answer does not establish an assistant's reliability.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/do-ai-assistants-give-the-same-answer-twice-we-asked-20-questions-three-times", "markdown": "https://wpnews.pro/news/do-ai-assistants-give-the-same-answer-twice-we-asked-20-questions-three-times.md", "text": "https://wpnews.pro/news/do-ai-assistants-give-the-same-answer-twice-we-asked-20-questions-three-times.txt", "jsonld": "https://wpnews.pro/news/do-ai-assistants-give-the-same-answer-twice-we-asked-20-questions-three-times.jsonld"}}