Public opinion polling often highlights the extent to which people are unsure about an issue. Knowing and quantifying that ambiguity is essential to any understanding of how important public debates may be evolving. Similarly, knowledge questions in polls can help provide valuable insights into the public’s familiarity with a topic – for example, whether people generally understand the digital privacy landscape or know key facts about the U.S. system of governance.
In contrast to humans, large language models are armed with vast quantities of training data drawn from around the internet. As such, there is reason to think that synthetic surveys will overstate the level of factual knowledge the typical American possesses.
These models also have a known tendency to want to provide answers to questions and will confidently provide incorrect answers. Because of this, we also might expect them to be consistently less willing than humans to say “I’m not sure” on surveys.
In our examination of synthetic polling, we found ample evidence that AI models do exactly these things.
This analysis is part of a larger evaluation of AI-generated synthetic samples in public opinion research. Read a summary of the main findingsand refer to the methodologyfor more details on how we conducted our synthetic poll and compared it with real survey results.
‘Not sure’ response options
We provide human survey-takers on the American Trends Panel (ATP) with an explicit “not sure” option on many questions we ask. Depending on how the questions are designed, this response may indicate uncertainty or indecision, or a lack of factual knowledge on a given topic.
In the instructions to our synthetic survey-takers, we made clear that it was realistic for them not to know about certain topics or to be incorrect at times about factual matters. But despite this prompting, our synthetic respondents chose “not sure” options far less often than real humans do.
Across all opinion questions where “not sure” was provided as an explicit option:
- Human panelists selected it 16% of the time.
- Synthetic respondents did so just 4% of the time.
In other words, the typical human respondent says they are not sure roughly four times as often as the typical synthetic respondent does when asked to share an opinion.
Questions testing factual knowledge
The ATP survey waves we chose to replicate with a synthetic sample included 13 questions testing factual knowledge on topics including the U.S. Constitution and the NATO alliance.
Across these questions, the synthetic panel tended to be far more “knowledgeable” than our human panel. Our human respondents answered correctly around half the time, on average, and there was no question that more than three-quarters answered correctly.
By contrast, our synthetic respondents answered correctly about 80% of the time. And on six different questions, the model estimated that 98% or more of the public knew the right answer.
In cases where the model decided that its persona would not know the correct answer to a question, it almost always indicated it was not sure. That is, our synthetic respondents almost never selected a factually incorrect answer choice.