Without search, even the best AI models get 1 in 4 answers wrong

Top AI models answer roughly 1 in 4 specific-fact questions incorrectly when they cannot search, according to Artificial Analysis data cited by Randy Olson. The AA-Omniscience dataset covers 6,000 questions on specific facts across 42 work topics, answered without search, with wrong answers counted as a share of answers given and declined questions excluded. Olson advised that models look facts up when accuracy matters, and noted the chart was made with evident-charts, his open source agent skill for explanatory charts.

Randy Olson on X: "Question 2 from @ArtificialAnlys data: how often do top AI models get specific facts wrong without search? Even the best get about 1 in 4 answers wrong. If a fact matters, have the model look it up. Made with evident-charts, my open source agent skill for explanatory charts." Question 2 from @ArtificialAnlys data: how often do top AI models get specific facts wrong without search? Even the best get about 1 in 4 answers wrong. If a fact matters, have the model look it up. Made with evident-charts, my open source agent skill for explanatory charts. Question 2 from @ArtificialAnlys data: how often do top AI models get specific facts wrong without search? Even the best get about 1 in 4 answers wrong. If a fact matters, have the model look it up. Made with evident-charts, my open source agent skill for explanatory charts. Data: AA-Omniscience, 6,000 questions about specific facts across 42 work topics, answered without search. Wrong answers as a share of answers given; declined questions don't count. Get the evident-charts agent skill here: