{"slug": "how-synthetic-respondents-express-certainty-and-factual-knowledge", "title": "How synthetic respondents express certainty and factual knowledge", "summary": "Pew Research Center found that synthetic survey respondents built on large language models selected explicit \"not sure\" options just 4% of the time, versus 16% for human panelists on its American Trends Panel, and answered factual knowledge questions correctly about 80% of the time versus roughly half for humans. Across 13 knowledge questions on topics including the U.S. Constitution and NATO, the models estimated 98% or more of the public knew the right answer on six questions, and synthetic respondents almost never chose a factually incorrect answer. The findings, part of Pew's broader evaluation of AI-generated synthetic samples in public opinion research, indicate synthetic polls overstate public factual knowledge and understate uncertainty.", "body_md": "Public opinion polling often highlights the extent to which people are unsure about an issue. Knowing and quantifying that ambiguity is essential to any understanding of how important public debates may be evolving.\n\nSimilarly, knowledge questions in polls can help provide valuable insights into the public’s familiarity with a topic – for example, whether people generally [understand the digital privacy landscape](https://www.pewresearch.org/internet/2023/08/17/what-americans-know-about-ai-cybersecurity-and-big-tech/) or know key facts about the [U.S. system of governance](https://www.pewresearch.org/short-reads/2023/11/07/what-americans-know-about-their-government/).\n\nIn contrast to humans, large language models are armed with vast quantities of training data drawn from around the internet. As such, there is reason to think that synthetic surveys will overstate the level of factual knowledge the typical American possesses.\n\nThese models also have a known tendency to [want to provide answers](https://medium.com/@markus_brinsa/why-ai-models-always-answer-even-when-they-shouldnt-e95081e3f46b) to questions and will [confidently provide incorrect answers](https://www.cmu.edu/dietrich/news/news-stories/2025/trent-cash-ai-overconfidence). Because of this, we also might expect them to be consistently less willing than humans to say “I’m not sure” on surveys.\n\nIn our examination of synthetic polling, we found ample evidence that AI models do exactly these things.\n\n*This analysis is part of a larger evaluation of AI-generated synthetic samples in public opinion research. Read* [a summary of the main findings](https://www.pewresearch.org/data-labs/2026/09/30/can-ai-stand-in-for-human-survey-takers-not-really/)*and refer to the* [methodology](https://www.pewresearch.org/data-labs/2026/09/30/methodology-silicon-samples/)*for more details on how we conducted our synthetic poll and compared it with real survey results.*\n\n### ‘Not sure’ response options\n\nWe provide human survey-takers on the [American Trends Panel](https://www.pewresearch.org/the-american-trends-panel/) (ATP) with an explicit “not sure” option on many questions we ask. Depending on how the questions are designed, this response may indicate uncertainty or indecision, or a lack of factual knowledge on a given topic.\n\nIn the instructions to our synthetic survey-takers, we made clear that it was realistic for them not to know about certain topics or to be incorrect at times about factual matters. But despite this prompting, our synthetic respondents chose “not sure” options far less often than real humans do.\n\nAcross all opinion questions where “not sure” was provided as an explicit option:\n\n- Human panelists selected it 16% of the time.\n- Synthetic respondents did so just 4% of the time.\n\nIn other words, the typical human respondent says they are not sure roughly four times as often as the typical synthetic respondent does when asked to share an opinion.\n\n### Questions testing factual knowledge\n\nThe ATP survey waves we chose to replicate with a synthetic sample included 13 questions testing factual knowledge on topics including the U.S. Constitution and the NATO alliance.\n\nAcross these questions, the synthetic panel tended to be far more “knowledgeable” than our human panel. Our human respondents answered correctly around half the time, on average, and there was no question that more than three-quarters answered correctly.\n\nBy contrast, our synthetic respondents answered correctly about 80% of the time. And on six different questions, the model estimated that 98% or more of the public knew the right answer.\n\nIn cases where the model decided that its persona would *not* know the correct answer to a question, it almost always indicated it was not sure. That is, our synthetic respondents almost never selected a factually incorrect answer choice.", "url": "https://wpnews.pro/news/how-synthetic-respondents-express-certainty-and-factual-knowledge", "canonical_source": "https://www.pewresearch.org/data-labs/2026/09/30/how-synthetic-respondents-express-certainty-and-factual-knowledge/", "published_at": "2026-09-30 17:55:19+00:00", "updated_at": "2026-09-30 18:19:53.383967+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research"], "entities": ["Pew Research Center", "American Trends Panel", "NATO", "U.S. Constitution"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-synthetic-respondents-express-certainty-and-factual-knowledge", "markdown": "https://wpnews.pro/news/how-synthetic-respondents-express-certainty-and-factual-knowledge.md", "text": "https://wpnews.pro/news/how-synthetic-respondents-express-certainty-and-factual-knowledge.txt", "jsonld": "https://wpnews.pro/news/how-synthetic-respondents-express-certainty-and-factual-knowledge.jsonld"}}