Ask an AI assistant the same question twice and you may get two different answers. In PlaceBench, our test of six assistants on everyday questions about real places, we asked 20 of the 150 questions three times each to see how often the answer holds.
| Assistant | Right on the first try | Right all 3 times | Grade changed between tries | Wrong at least once |
|---|---|---|---|---|
| Dots | 18 of 20 | 17 | 3 | 0 |
| Claude Duet | 17 | 13 | 6 | 4 |
| Muse | 15 | 12 | 7 | 2 |
| Ask Maps | 14 | 12 | 5 | 5 |
| Instinct | 14 | 5 | 15 | 7 |
| Grok Bot | 13 | 8 | 11 | 6 |
"Grade changed" counts the questions where the three answers didn't all get the same grade. We computed the last two columns from the public grades file.
The methodology also reports pass^k, the chance that k tries at a question are all right. For three tries: Dots 85%, Claude Duet 65%, Muse 60%, Ask Maps 60%, Grok Bot 40%, Instinct 25%.
If you build on top of an assistant, one good answer in testing isn't enough. Ask the same question several times, or check the facts the answer depends on, such as whether the place is open at that hour. 20 questions is a small sample. Instinct keeps one continuous conversation and Dots keeps memory between chats, so their repeats can draw on earlier answers. We fixed the 20 questions before collecting any answers.
Questions, answer keys, sources and every grade, under CC BY 4.0: github.com/voygr-tech/placebench. Full write-up: voygr.tech/placebench.