Large language models can predict the results of social science experiments

Researchers built an archive of 70 preregistered survey experiments and found that GPT-4 can predict the outcomes of social science experiments with accuracy similar to pooled human forecasts, though it systematically overestimates effect sizes. The study suggests large language models can augment experimental methods while raising concerns about bias and misuse.

Abstract There is growing interest in how large language models LLMs can advance social and behavioural science 1,2,3,4,5. Previous work has assessed LLMs’ ability to predict survey responses , but less is known about whether they can predict the outcomes of social science experiments 6 ref-CR6 , 7 ref-CR7 , 8 ref-CR8 , 9 /articles/s41586-026-10742-x ref-CR9 , particularly those absent from training data. Here we built an archive of 70 preregistered, nationally representative survey experiments in the USA involving 469 experimental effects and 119,330 participants. We prompted an LLM to simulate how representative samples from American individuals would respond to experimental stimuli, and then we inferred treatment effects by comparing simulated responses across conditions. Predictions derived from GPT-4, whose training-data cutoff predated the publication of many studies in our archive, were strongly correlated with actual treatment effects, achieving accuracy similar to pooled human forecasts. Correlations remained high for studies not published or publicly posted by the model’s training-data cutoff date and for predictions from prominent open-weight models. Despite high correlations, predictions systematically overestimated effect sizes. In a secondary archive of 15 megastudies featuring 606 effects, correlations were lower but comparable to those of pooled expert forecasters. To assess implications for scientific practice, we surveyed 460 social scientists about probable uses and perceived risks and used our archives to assess several applications pilot testing, intervention selection, identifying effects needing replication and risks bias, misuse . Together, these results indicate that LLMs can augment experimental methods in science and practice while raising important considerations for responsible use. 10 /articles/s41586-026-10742-x ref-CR10 This is a preview of subscription content, access via your institution https://wayf.springernature.com?redirect uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fs41586-026-10742-x Access options Access Nature and 54 other Nature Portfolio journals Get Nature+, our best-value online-access subscription 27,99 € / 30 days cancel any time Subscribe to this journal Receive 52 print issues and online access 199,00 € per year only 3,83 € per issue Buy this article - Purchase on SpringerLink - Instant access to the full article PDF. 39,95 € Prices may be subject to local taxes which are calculated during checkout Data availability All data used in this paper are available at https://codeocean.com/capsule/9843791/tree/v1 https://codeocean.com/capsule/9843791/tree/v1 . All study materials are available in the Supplementary Information /articles/s41586-026-10742-x MOESM1 . Source data /articles/s41586-026-10742-x Sec26 are provided with this paper. Code availability Code to reproduce the results is available at https://codeocean.com/capsule/9843791/tree/v1 https://codeocean.com/capsule/9843791/tree/v1 . We also invite researchers to try the model in our web demo https://treatmenteffect.app/ https://treatmenteffect.app/ , which generates AI-based forecasts of experimental effects using the approach detailed in this paper. References Bail, C. A. Can generative AI improve social science? Proc. Natl Acad. Sci. USA 121 , e2314021121 2024 .Grossmann, I. et al. AI and the transformation of social science research. Science 380 , 1108–1109 2023 .Luo, X. et al. Large language models surpass human experts in predicting neuroscience results. Nat. Hum. Behav. 9 , 305–315 2025 .Crockett, M. & Messeri, L. Should large language models replace human participants? Preprint at PsyArXiv https://doi.org/10.31234/osf.io/4zdx9 https://doi.org/10.31234/osf.io/4zdx9 2024 .Abdurahman, S. et al. Perils and opportunities in using large language models in psychological research. PNAS Nexus 3 , pgae245 2024 .Argyle, L. P. et al. Out of one, many: using language models to simulate human samples. Polit. Anal. 31 , 337–351 2023 .Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B. & Larson, J. M. Synthetic replacements for human survey data? The perils of large language models. Polit. Anal. 32 , 401–416 2024 .Atari, M., Xue, M. J., Park, P. S., Blasi, D. & Henrich, J. Which humans? Preprint at PsyArXiv https://doi.org/10.31234/osf.io/5b26t https://doi.org/10.31234/osf.io/5b26t 2023 .Dillion, D., Tandon, N., Gu, Y. & Gray, K. Can AI language models replace human participants? Trends Cogn. Sci. 27 , 597–600 2023 .Filippas, A., Horton, J. J. & Manning, B. S. Large language models as simulated economic agents: what can we learn from homo silicus? In Proc. 25th ACM Conference on Economics and Computation 614–615 ACM, 2024 .Binz, M. & Schulz, E. Using cognitive psychology to understand GPT-3. Proc. Natl Acad. Sci. USA 120 , e2218523120 2023 .Bai, H., Voelkel, J. G., Muldowney, S., Eichstaedt, J. C. & Willer, R. LLM-generated messages can persuade humans on policy issues. Nat. Commun. 16 , 6037 2025 .Hackenburg, K. et al. The levers of political persuasion with conversational artificial intelligence. Science 390 , eaea3884 2025 .Meta Fundamental AI Research Diplomacy Team FAIR et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science 378 , 1067–1074 2022 .Park, J. S. et al. Generative agents: interactive simulacra of human behavior. In Proc. 36th Annual ACM Symposium on User Interface Software and Technology eds Follmer, S. et al. 1–22 ACM, 2023 .Schramowski, P., Turan, C., Andersen, N., Rothkopf, C. A. & Kersting, K. Large pre-trained language models contain human-like biases of what is right and wrong to do. Nat. Mach. Intell. 4 , 258–268 2022 .Yin, Y., Jia, N. & Wakslak, C. J. AI can help people feel heard, but an AI label diminishes this impact. Proc. Natl Acad. Sci. USA 121 , e2319112121 2024 .Lehr, S. A., Caliskan, A., Liyanage, S. & Banaji, M. R. ChatGPT as research scientist: probing GPT’s capabilities as a research librarian, research ethicist, data generator, and data predictor. Proc. Natl Acad. Sci. USA 121 , e2404328121 2024 .Time-sharing Experiments for the Social Sciences. TESS www.tessexperiments.org http://www.tessexperiments.org 2026 .Coppock, A., Leeper, T. J. & Mullinix, K. J. Generalizability of heterogeneous treatment effect estimates across samples. Proc. Natl Acad. Sci. USA 115 , 12441–12446 2018 .Chang, S., Kennedy, A., Leonard, A. & List, J. A. 12 Best Practices for Leveraging Generative AI in Experimental Research National Bureau of Economic Research, 2024 .Broska, D., Howes, M. & van Loon, A. The mixed subjects design: treating large language models as potentially informative observations. Sociol. Methods Res. 54 , 1074–1109 2025 .Chu, J. Y. et al. Academics are more specific, and practitioners more sensitive, in forecasting interventions to strengthen democratic attitudes. Proc. Natl Acad. Sci. USA 121 , e2307008121 2024 .DellaVigna, S. & Linos, E. RCTs to scale: comprehensive evidence from two nudge units. Econometrica 90 , 81–116 2022 .Dreber, A. et al. Using prediction markets to estimate the reproducibility of scientific research. Proc. Natl Acad. Sci. USA 112 , 15343–15347 2015 .Milkman, K. L. et al. Megastudies improve the impact of applied behavioural science. Nature 600 , 478–483 2021 .Harding, J., D’Alessandro, W., Laskowski, N. G. & Long, R. AI language models cannot replace human research participants. AI Soc. 39 , 2603–2605 2024 .Messeri, L. & Crockett, M. J. Artificial intelligence and illusions of understanding in scientific research. Nature 627 , 49–58 2024 .Binz, M. et al. A foundation model to predict and capture human cognition. Nature 644 , 1002–1009 2025 .Petrov, N. B., Serapio-García, G. & Rentfrow, J. Limited ability of LLMs to simulate human psychological behaviours: a psychometric analysis. Preprint at https://arxiv.org/abs/2405.07248 https://arxiv.org/abs/2405.07248 2024 .Tjuatja, L., Chen, V., Wu, T., Talwalkwar, A. & Neubig, G. Do LLMs exhibit human-like response biases? A case study in survey design. Trans. Assoc. Comput. Linguist. 12 , 1011–1026 2024 .Chen, Y., Hu, Y. & Lu, Y. Predicting field experiments with large language models. Preprint at https://arxiv.org/abs/2504.01167 https://arxiv.org/abs/2504.01167 2025 .Wang, X. et al. Self-consistency improves chain of thought reasoning in language models. In Proc. 11th International Conference on Learning Representations poster ICLR, 2023 .Arora, S. et al. Ask me anything: a simple strategy for prompting language models. In Proc. 11th International Conference on Learning Representations ICLR, 2023 .Voronov, A., Wolf, L. & Ryabinin, M. Mind your format: towards consistent evaluation of in-context learning improvements. In Findings of the Association for Computational Linguistics: ACL 2024 eds Ku, L.-W. et al. 6287–6310 ACL, 2024 .Hou, B., O’Connor, J., Andreas, J., Chang, S. & Zhang, Y. Promptboosting: black-box text classification with ten forward passes. In Proc. International Conference on Machine Learning eds Krause, A. et al. 13309–13324 PMLR, 2023 .Open Science Collaboration. Estimating the reproducibility of psychological science. Science 349 , aac4716 2015 .Milkman, K. L. et al. Megastudy shows that reminders boost vaccination but adding free rides does not. Nature 631 , 179–188 2024 .Milkman, K. L. et al. A 680,000-person megastudy of nudges to encourage vaccination in pharmacies. Proc. Natl Acad. Sci. USA 119 , e2115126119 2022 .Zickfeld, J. H. et al. Effectiveness of ex ante honesty oaths in reducing dishonesty depends on content. Nat. Hum. Behav. 9 , 169–187 2025 .Broockman, D. E., Kalla, J. L., Caballero, C. & Easton, M. Political practitioners poorly predict which messages persuade the public. Proc. Natl Acad. Sci. USA 121 , e2400076121 2024 .DellaVigna, S. & Vivalt, E. Forecasting social science: evidence from 100 projects. National Bureau of Economic Research https://doi.org/10.3386/w34493 https://doi.org/10.3386/w34493 2025 .Allen, J., Watts, D. J. & Rand, D. G. Quantifying the impact of misinformation and vaccine-skeptical content on Facebook. Science 384 , eadk3451 2024 .DellaVigna, S. & Pope, D. What motivates effort? Evidence and expert forecasts. Rev. Econ. Stud. 85 , 1029–1069 2018 .Vlasceanu, M. et al. Addressing climate change with behavioral science: a global intervention tournament in 63 countries. Sci. Adv. 10 , eadj5778 2024 .Tappin, B. M., Wittenberg, C., Hewitt, L. B., Berinsky, A. J. & Rand, D. G. Quantifying the potential persuasive returns to political microtargeting. Proc. Natl Acad. Sci. USA 120 , e2216261120 2023 .Voelkel, J. G. et al. Megastudy testing 25 treatments to reduce antidemocratic attitudes and partisan animosity. Science 386 , eadh4764 2024 .Voelkel, J. G. et al. A registered report megastudy on the persuasiveness of the most-cited climate messages. Nat. Clim. Chang. 16 , 214–225 2026 .Saccardo, S. et al. Field testing the transferability of behavioural science knowledge on promoting vaccinations. Nat. Hum. Behav. 8 , 878–890 2024 .Mason, K. et al. A megastudy of behavioral interventions to increase voter registration ahead of the 2024 US presidential election. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/p6b2e v3 https://doi.org/10.31234/osf.io/p6b2e v3 2025 .Goldwert, D. et al. A megastudy of behavioral interventions to catalyze public, political, and financial climate advocacy. PNAS Nexus 5 , pgaf400 2026 .Druckman, J. N. Experimental Thinking Cambridge Univ. Press, 2022 .Freese, J. & Peterson, D. Replication in social science. Annu. Rev. Sociol. 43 , 147–165 2017 .Bai, X., Wang, A., Sucholutsky, I. & Griffiths, T. L. Explicitly unbiased large language models still form biased associations. Proc. Natl Acad. Sci. USA 122 , e2416228122 2025 .Wang, A., Morgenstern, J. & Dickerson, J. P. Large language models that replace human participants can harmfully misportray and flatten identity groups. Nat. Mach. Intell. 7 , 400–411 2025 .Park, P. S., Schoenegger, P. & Zhu, C. Diminished diversity-of-thought in a standard large language model. Behav. Res. Methods 56 , 5754–5770 2024 .Frank, M. C. Openly accessible LLMs can help us to understand human cognition. Nat. Hum. Behav. 7 , 1825–1827 2023 .Wack, M., Ehrett, C., Linvill, D. & Warren, P. Generative propaganda: evidence of AI’s impact from a state-backed disinformation campaign. PNAS Nexus 4 , pgaf083 2025 .Response retrieved from ChatGPT-4 on 22 May. ChatGPT https://chatgpt.com/share/2a2b66d0-e9fd-421f-bedd-bb6f9354742a https://chatgpt.com/share/2a2b66d0-e9fd-421f-bedd-bb6f9354742a 2024 .Almaatouq, A. et al. Beyond playing 20 questions with nature: integrative experiment design in the social and behavioral sciences. Behav. Brain Sci. 47 , e33 2024 .Zhu, J.-Q., Xie, H., Arumugam, D., Wilson, R. C. & Griffiths, T. L. Using reinforcement learning to train large language models to explain human decisions. Preprint at https://arxiv.org/abs/2505.11614 https://arxiv.org/abs/2505.11614 2025 .Tyner, A. H. et al. Investigating the replicability of the social and behavioural sciences. Nature 652 , 143–150 2026 .Cremer, D. D. & Kasparov, G. AI should augment human intelligence, not replace it. Harvard Business Review https://hbr.org/2021/03/ai-should-augment-human-intelligence-not-replace-it https://hbr.org/2021/03/ai-should-augment-human-intelligence-not-replace-it 2021 .Cummins, J. The threat of analytic flexibility in using large language models to simulate human data: a call to attention. Preprint at https://arxiv.org/abs/2509.13397 https://arxiv.org/abs/2509.13397 2025 .Obermeyer, Z., Powers, B., Vogeli, C. & Mullainathan, S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366 , 447–453 2019 .Coppock, A. Generalizing from survey experiments conducted on mechanical turk: a replication approach. Political Sci. Res. Methods 7 , 613–628 2019 .Mullinix, K. J., Leeper, T. J., Druckman, J. N. & Freese, J. The generalizability of survey experiments. J. Exp. Political Sci. 2 , 109–138 2015 .Achiam, J. et al. GPT-4 technical report. Preprint at https://arxiv.org/abs/2303.08774 https://arxiv.org/abs/2303.08774 2023 .Viechtbauer, W. metafor: Meta-Analysis Package for R v.4.8-0 2015 . Rios, K., Roth, Z. C. & Coleman, T. J. III. The importance of scientists’ intellectual humility for communicating effectively across ideological and identity-based divides. Proc. Natl Acad. Sci. USA 122 , e2400930121 2025 . Acknowledgements We thank N. Egan, K. Fuller, V. Isarraras, S. Kane, B. Kinder, A. Kissi, K. McPeek, S. Muldowney, C. Redekopp and T. Ren for research assistance. We are grateful to researchers who shared data, including J. Chu, S. DellaVigna, J. Druckman, A. Duckworth, D. Goldwert, N. Malhotra, K. Mason, K. Milkman, K. Okst, S. Saccardo, M. Vlasceanu and J. Voelkel. We also thank J. Allen, M. Atari, C. Bail, M. Bernstein, M. Dehghani, D. Broockman, K. Clayton, M. Crockett, D. Ganguli, D. Dillon, J. Eichstaedt, M. Frank, N. Goodman, D. Green, J. Kalla, P. Liang, J. McClelland, D. Rand, R. Orchinik, N. Otis, S. Rathje, B. Tappin, A. Van Loon, A. Waytz and D. Willner for invaluable feedback on the project. Funding We acknowledge funding from the Stanford Center on Philanthropy and Civil Society. Author information Authors and Affiliations Contributions L.H., A.A. and R.W. designed the research. A.A., L.H., I.G. and R.W. assembled the experimental archives. L.H. and A.A. analysed the data. A.A., L.H. and R.W. wrote the paper. L.H. built the web tool. Corresponding authors Ethics declarations Competing interests The authors declare no competing interests. Peer review Peer review information Nature thanks Jamie Cummins, James Druckman, John Protzko and Youyou Wu for their contribution to the peer review of this work. Peer reviewer reports /articles/s41586-026-10742-x MOESM3 are available. Additional information Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Extended data figures and tables Extended Data Fig. 1 Method. /articles/s41586-026-10742-x/figures/4 We re-analyze raw data from 70 nationally-representative U.S. studies and estimate treatment effects in a consistent manner. We use a Large Language Model to predict results of those same experiments, providing the original study materials and demographically-diverse participant profiles, and then calculate the average predicted response for each condition. We evaluate the accuracy of the model in terms of the correspondence between observed- and predicted- treatment effects. Extended Data Fig. 2 Averaging across more LLM prompts improves predictive accuracy. /articles/s41586-026-10742-x/figures/5 Error bars indicate 95% confidence intervals for the mean Pearson correlation across 32 simulation runs. Each run uses a different random subset of prompts drawn without replacement from 120 . The CI reflects uncertainty in the mean across simulation runs, but does not include uncertainty over the sample of 70 experiments or the sample of 120 prompts. Extended Data Fig. 3 The accuracy of LLM-derived predictions for demographic subgroups in the US. /articles/s41586-026-10742-x/figures/6 Error bars indicate 95% confidence intervals. Supplementary information Supplementary Information download PDF https://static-content.springer.com/esm/art%3A10.1038%2Fs41586-026-10742-x/MediaObjects/41586 2026 10742 MOESM1 ESM.pdf Supplementary Figs. 1–10, Tables 1–6 and text on the construction of the primary and secondary experimental archives, prompting strategy, layperson and expert forecasts, survey recruitment, analytical methods for both archives and the scientific applications, robustness checks and further applications and risk analyses. Supplementary Figs. 1–10 and Tables 1–6 provide additional detail on the study archives and methods, assess the robustness of LLM-based predictions, present further analyses of pilot-testing and intervention-selection use cases, and report detailed survey results from social scientists. Rights and permissions Springer Nature or its licensor e.g. a society or other partner holds exclusive rights to this article under a publishing agreement with the author s or other rightsholder s ; author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law. About this article Cite this article Ashokkumar, A., Hewitt, L., Ghezae, I. et al. Large language models can predict the results of social science experiments. Nature 2026 . https://doi.org/10.1038/s41586-026-10742-x Received: Accepted: Published: Version of record: DOI: https://doi.org/10.1038/s41586-026-10742-x