Do AI assistants give the same answer twice? We asked 20 questions three times A benchmark called PlaceBench tested six AI assistants on 20 everyday questions about real places, asking each question three times, and found that answer consistency varied widely: Dots answered 17 of 20 correctly on all three tries, while Instinct managed only 5. The team reports pass^3 scores ranging from 85% for Dots to 25% for Instinct, and notes that assistants with persistent memory or continuous conversations can draw on earlier answers when repeating a question. The questions, answer keys, sources and grades are published under CC BY 4.0. Ask an AI assistant the same question twice and you may get two different answers. In PlaceBench, our test of six assistants on everyday questions about real places, we asked 20 of the 150 questions three times each to see how often the answer holds. | Assistant | Right on the first try | Right all 3 times | Grade changed between tries | Wrong at least once | |---|---|---|---|---| | Dots | 18 of 20 | 17 | 3 | 0 | | Claude Duet | 17 | 13 | 6 | 4 | | Muse | 15 | 12 | 7 | 2 | | Ask Maps | 14 | 12 | 5 | 5 | | Instinct | 14 | 5 | 15 | 7 | | Grok Bot | 13 | 8 | 11 | 6 | "Grade changed" counts the questions where the three answers didn't all get the same grade. We computed the last two columns from the public grades file. The methodology also reports pass^k, the chance that k tries at a question are all right. For three tries: Dots 85%, Claude Duet 65%, Muse 60%, Ask Maps 60%, Grok Bot 40%, Instinct 25%. If you build on top of an assistant, one good answer in testing isn't enough. Ask the same question several times, or check the facts the answer depends on, such as whether the place is open at that hour. 20 questions is a small sample. Instinct keeps one continuous conversation and Dots keeps memory between chats, so their repeats can draw on earlier answers. We fixed the 20 questions before collecting any answers. Questions, answer keys, sources and every grade, under CC BY 4.0: github.com/voygr-tech/placebench https://github.com/voygr-tech/placebench . Full write-up: voygr.tech/placebench https://voygr.tech/placebench/ .