Synthetic surveys and ‘timely and topical’ questions

Pew Research Center found that AI-generated synthetic surveys failed to track real shifts in U.S. public opinion, estimating President Donald Trump's approval at a flat 46% in both January and April 2026 while its American Trends Panel of real adults recorded a decline from 37% to 34%. The synthetic samples missed the drop in approval among Republicans and Republican leaners by as much as 19 percentage points, and Pew said the model often failed to reflect opinion shaped by cultural or economic circumstances arising after its knowledge cutoff. The findings come from Pew's broader evaluation of AI-generated synthetic samples in public opinion research, covering more than 120 questions across three surveys.

Public opinion often seems slow to change, but the opinions and attitudes of the public can sometimes shift quickly and dramatically in response to current events. This is part of what’s interesting and challenging about studying public opinion. Sometimes people change their minds relatively rapidly, as happened with former President George W. Bush’s increased approval rating https://www.pewresearch.org/politics/2008/12/18/bush-and-public-opinion/ in the wake of 9/11. At other times, segments of the population can switch places on issues, as happened with Democrats’ and Republicans’ perceptions of the social media platform X formerly Twitter https://www.pewresearch.org/short-reads/2025/06/05/republicans-and-democrats-on-x-differ-over-the-sites-politics-and-their-experiences/ after Elon Musk purchased the company. One appeal of AI polling is its speed. At least in theory, these polls can be fielded nearly instantly without the need to assemble a representative sample of humans who are willing to take the survey. At the same time, other ways of inferring public opinion without directly measuring it through polling like statistical modeling can struggle to identify changes in sentiment when they happen quickly, such as in response to breaking news. We found that polling with AI models is no different. In our analysis comparing surveys of AI respondents with surveys of real humans, there were numerous instances in which the model failed to accurately reflect public opinion in unique cultural or economic circumstances. That is especially – but not always – true when those circumstances developed after the model’s knowledge cutoff. 2 fn-574097-2 This analysis is part of a larger evaluation of AI-generated synthetic samples in public opinion research. Read a summary of the main findings https://www.pewresearch.org/data-labs/2026/09/30/can-ai-stand-in-for-human-survey-takers-not-really/ and refer to the methodology https://www.pewresearch.org/data-labs/2026/09/30/methodology-silicon-samples/ for more details on how we conducted our synthetic poll and compared it with real survey results. Using synthetic polls to reproduce ongoing trends in public opinion Pew Research Center polled Americans on their approval of President Donald Trump using the American Trends Panel https://www.pewresearch.org/the-american-trends-panel/ ATP seven times between the beginning of 2025 and May 2026, including on two surveys that we replicated with synthetic respondents. In January 2026, 37% of U.S. adults approved of Trump’s job performance. By April, when the next replicated survey with this question was fielded, that had fallen to 34%. But our synthetic samples estimated Trump’s approval at a steady 46% for both waves. The last time Trump’s approval was actually that high in our polling was at the beginning of his second term in February 2025, when 47% of U.S. adults said they approved of the job he was doing. Not only did the synthetic replication in January 2026 fail to capture the decline in approval that had occurred since Trump’s return to office, but when we repeated the exercise in April, it gave us the exact same topline number. If we look at the partisan breakdown on presidential approval over time, the story has been remarkably consistent for Democrats and Democratic leaners; the synthetic survey replicated their approval numbers quite closely. The biggest shift in attitudes in our human surveys has been a decline among Republicans and Republican leaners; our AI poll missed the mark for this group by as much as 19 percentage points. Presidential approval is an especially long-standing trend question that the AI poll overestimated by a considerable margin in this case. But on its own, the fact that a question is trendable doesn’t tell us very much about the accuracy we should expect from an AI replication. The three surveys we evaluated in this study contained more than 120 questions we had asked at least once in the past. Collectively, these trend questions had an average error rate of 12 percentage points, regardless of how much opinion had actually changed over the life of the trend. That’s the same as the overall average error across all three survey waves. Using synthetic polls to measure public opinion on timely and topical issues In addition to measuring how the model does at replicating some long-standing trends, we also looked at some new questions that asked about contemporaneous events the model might not have context for in its training data. These serve as examples of how synthetic polls can fail to capture how people feel about an issue when current sentiments break with past precedent in some way. Here are some “timely and topical” issues where our model failed to capture the state of public sentiment. Immigration enforcement U.S. Immigration and Customs Enforcement’s actions, such as “ Operation Metro Surge https://www.npr.org/2026/02/12/nx-s1-5712280/minnesota-ice-surge-ends ” in winter 2025-26, generated public controversy over a range of issues, including federal agents’ wearing of masks or other face coverings to hide their identities while working. Overall, the synthetic sample greatly underestimated the share of Americans who say it’s acceptable for immigration officers to cover their faces. Among Republicans and Republican leaners, the difference was even starker: 35% of synthetic Republicans find it acceptable, but the actual share among Republicans is nearly double that 67% . Military conflict in Iran Our synthetic poll consistently overstated public support for the conflict between the United States, Israel and Iran that began on Feb. 28, 2026. The model’s estimates were 10 to 19 percentage points higher than human respondents on topics ranging from overall approval of the strikes against Iran to whether the conflict will make the Iranian people better or worse off. On this issue, too, the model was especially prone to overstating support among Republicans. In our synthetic panel, around nine-in-ten Republicans and Republican leaners approve of the strikes against Iran; are confident Trump can make good decisions about Iran policy; think the strikes were the right decision; and say the U.S. is doing enough to prevent civilian casualties. In reality, around two-thirds of Republicans hold each of these views. That is a solid majority, but hardly the unanimous agreement implied by the synthetic panel. The model did a better job at estimating disapproval of the conflict among Democrats and Democratic leaners. But even among this group there were notable misses. For instance, 94% of Democrats in our AI poll think the conflict will make the Iranian people worse off, while in reality, that figure is 56%. Data centers In a January 2026 survey, we asked Americans about their views on data centers – a newly salient political issue. At the time, public awareness of data centers was quite varied: 25% of U.S. adults had heard a lot about them, but a nearly identical share had heard nothing at all. Around half of Americans were in the middle, reporting that they had heard a little about data centers. By contrast, our synthetic poll estimated that 94% of U.S. adults have heard a little about data centers, and just 3% each have heard a lot or nothing at all. In a series of follow-up questions asked of both human and AI respondents who had heard about data centers, nearly every synthetic respondent said they weren’t sure if there were any data centers built or planned in their area – more than double the 44% of human respondents who were similarly unsure. The AI poll also missed badly on questions about the perceived impacts of data centers. At least 97% of the synthetic respondents who received these questions said data centers are mostly bad for home energy costs, the environment and the quality of life for people who live nearby. Although many humans hold these views, the actual shares among those who have heard about data centers range from 41% to 53%. At the same time, the AI poll greatly understated the share of the public who says that data centers are mostly bad for local jobs or tax revenue. Economic concerns In a series of questions about economic challenges facing Americans today, the synthetic sample overestimated certain concerns while underestimating others. In particular, the model estimated that just 23% of Americans are very concerned about the price of electricity. The actual share 52% is more than double that.