cd /news/artificial-intelligence/do-ai-assistants-give-the-same-answe… · home › topics › artificial-intelligence › article
[ARTICLE · art-173172] src=dev.to ↗ pub= topic=artificial-intelligence verified=true sentiment=· neutral

Do AI assistants give the same answer twice? We asked 20 questions three times

A benchmark called PlaceBench tested six AI assistants on 20 everyday questions about real places, asking each question three times, and found that answer consistency varied widely: Dots answered 17 of 20 correctly on all three tries, while Instinct managed only 5. The team reports pass^3 scores ranging from 85% for Dots to 25% for Instinct, and notes that assistants with persistent memory or continuous conversations can draw on earlier answers when repeating a question. The questions, answer keys, sources and grades are published under CC BY 4.0.

recommendation

Repeat each question several times in testing and verify the underlying facts, since a single correct answer does not establish an assistant's reliability.

more in playbook: Evals and testing for LLM apps
by read1 min views2 publishedNov 20, 2026

Ask an AI assistant the same question twice and you may get two different answers. In PlaceBench, our test of six assistants on everyday questions about real places, we asked 20 of the 150 questions three times each to see how often the answer holds.

Assistant Right on the first try Right all 3 times Grade changed between tries Wrong at least once
Dots 18 of 20 17 3 0
Claude Duet 17 13 6 4
Muse 15 12 7 2
Ask Maps 14 12 5 5
Instinct 14 5 15 7
Grok Bot 13 8 11 6

"Grade changed" counts the questions where the three answers didn't all get the same grade. We computed the last two columns from the public grades file.

The methodology also reports pass^k, the chance that k tries at a question are all right. For three tries: Dots 85%, Claude Duet 65%, Muse 60%, Ask Maps 60%, Grok Bot 40%, Instinct 25%.

If you build on top of an assistant, one good answer in testing isn't enough. Ask the same question several times, or check the facts the answer depends on, such as whether the place is open at that hour. 20 questions is a small sample. Instinct keeps one continuous conversation and Dots keeps memory between chats, so their repeats can draw on earlier answers. We fixed the 20 questions before collecting any answers.

Questions, answer keys, sources and every grade, under CC BY 4.0: github.com/voygr-tech/placebench. Full write-up: voygr.tech/placebench.

── more in #artificial-intelligence 4 stories · sorted by recency
── more on @placebench 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/do-ai-assistants-giv…] indexed:0 read:1min 2026-11-20 · —