{"slug": "large-language-models-can-predict-the-results-of-social-science-experiments", "title": "Large language models can predict the results of social science experiments", "summary": "Researchers built an archive of 70 preregistered survey experiments and found that GPT-4 can predict the outcomes of social science experiments with accuracy similar to pooled human forecasts, though it systematically overestimates effect sizes. The study suggests large language models can augment experimental methods while raising concerns about bias and misuse.", "body_md": "## Abstract\n\nThere is growing interest in how large language models (LLMs) can advance social and behavioural science 1,2,3,4,5. Previous work has assessed LLMs’ ability to predict survey responses\n\n, but less is known about whether they can predict the outcomes of social science experiments\n\n[6](#ref-CR6),[7](#ref-CR7),[8](#ref-CR8),[9](/articles/s41586-026-10742-x#ref-CR9), particularly those absent from training data. Here we built an archive of 70 preregistered, nationally representative survey experiments in the USA involving 469 experimental effects and 119,330 participants. We prompted an LLM to simulate how representative samples from American individuals would respond to experimental stimuli, and then we inferred treatment effects by comparing simulated responses across conditions. Predictions derived from GPT-4, whose training-data cutoff predated the publication of many studies in our archive, were strongly correlated with actual treatment effects, achieving accuracy similar to pooled human forecasts. Correlations remained high for studies not published or publicly posted by the model’s training-data cutoff date and for predictions from prominent open-weight models. Despite high correlations, predictions systematically overestimated effect sizes. In a secondary archive of 15 megastudies featuring 606 effects, correlations were lower but comparable to those of pooled expert forecasters. To assess implications for scientific practice, we surveyed 460 social scientists about probable uses and perceived risks and used our archives to assess several applications (pilot testing, intervention selection, identifying effects needing replication) and risks (bias, misuse). Together, these results indicate that LLMs can augment experimental methods in science and practice while raising important considerations for responsible use.\n\n[10](/articles/s41586-026-10742-x#ref-CR10)This is a preview of subscription content, [access via your institution](https://wayf.springernature.com?redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fs41586-026-10742-x)\n\n## Access options\n\nAccess Nature and 54 other Nature Portfolio journals\n\nGet Nature+, our best-value online-access subscription\n\n27,99 € / 30 days\n\ncancel any time\n\nSubscribe to this journal\n\nReceive 52 print issues and online access\n\n199,00 € per year\n\nonly 3,83 € per issue\n\nBuy this article\n\n- Purchase on SpringerLink\n- Instant access to the full article PDF.\n\n39,95 €\n\nPrices may be subject to local taxes which are calculated during checkout\n\n## Data availability\n\nAll data used in this paper are available at [https://codeocean.com/capsule/9843791/tree/v1](https://codeocean.com/capsule/9843791/tree/v1). All study materials are available in the [Supplementary Information](/articles/s41586-026-10742-x#MOESM1). [Source data](/articles/s41586-026-10742-x#Sec26) are provided with this paper.\n\n## Code availability\n\nCode to reproduce the results is available at [https://codeocean.com/capsule/9843791/tree/v1](https://codeocean.com/capsule/9843791/tree/v1). We also invite researchers to try the model in our web demo ([https://treatmenteffect.app/](https://treatmenteffect.app/)), which generates AI-based forecasts of experimental effects using the approach detailed in this paper.\n\n## References\n\nBail, C. A. Can generative AI improve social science?\n\n*Proc. Natl Acad. Sci. USA***121**, e2314021121 (2024).Grossmann, I. et al. AI and the transformation of social science research.\n\n*Science***380**, 1108–1109 (2023).Luo, X. et al. Large language models surpass human experts in predicting neuroscience results.\n\n*Nat. Hum. Behav.***9**, 305–315 (2025).Crockett, M. & Messeri, L. Should large language models replace human participants? Preprint at\n\n*PsyArXiv*[https://doi.org/10.31234/osf.io/4zdx9](https://doi.org/10.31234/osf.io/4zdx9)(2024).Abdurahman, S. et al. Perils and opportunities in using large language models in psychological research.\n\n*PNAS Nexus***3**, pgae245 (2024).Argyle, L. P. et al. Out of one, many: using language models to simulate human samples.\n\n*Polit. Anal.***31**, 337–351 (2023).Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B. & Larson, J. M. Synthetic replacements for human survey data? The perils of large language models.\n\n*Polit. Anal.***32**, 401–416 (2024).Atari, M., Xue, M. J., Park, P. S., Blasi, D. & Henrich, J. Which humans? Preprint at\n\n*PsyArXiv*[https://doi.org/10.31234/osf.io/5b26t](https://doi.org/10.31234/osf.io/5b26t)(2023).Dillion, D., Tandon, N., Gu, Y. & Gray, K. Can AI language models replace human participants?\n\n*Trends Cogn. Sci.***27**, 597–600 (2023).Filippas, A., Horton, J. J. & Manning, B. S. Large language models as simulated economic agents: what can we learn from homo silicus? In\n\n*Proc. 25th ACM Conference on Economics and Computation*614–615 (ACM, 2024).Binz, M. & Schulz, E. Using cognitive psychology to understand GPT-3.\n\n*Proc. Natl Acad. Sci.**USA*** 120**, e2218523120 (2023).Bai, H., Voelkel, J. G., Muldowney, S., Eichstaedt, J. C. & Willer, R. LLM-generated messages can persuade humans on policy issues.\n\n*Nat. Commun.***16**, 6037 (2025).Hackenburg, K. et al. The levers of political persuasion with conversational artificial intelligence.\n\n*Science***390**, eaea3884 (2025).Meta Fundamental AI Research Diplomacy Team (FAIR) et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning.\n\n*Science***378**, 1067–1074 (2022).Park, J. S. et al. Generative agents: interactive simulacra of human behavior. In\n\n*Proc. 36th Annual ACM Symposium on User Interface Software and Technology*(eds Follmer, S. et al.) 1–22 (ACM, 2023).Schramowski, P., Turan, C., Andersen, N., Rothkopf, C. A. & Kersting, K. Large pre-trained language models contain human-like biases of what is right and wrong to do.\n\n*Nat. Mach. Intell.***4**, 258–268 (2022).Yin, Y., Jia, N. & Wakslak, C. J. AI can help people feel heard, but an AI label diminishes this impact.\n\n*Proc. Natl Acad. Sci.**USA*** 121**, e2319112121 (2024).Lehr, S. A., Caliskan, A., Liyanage, S. & Banaji, M. R. ChatGPT as research scientist: probing GPT’s capabilities as a research librarian, research ethicist, data generator, and data predictor.\n\n*Proc. Natl Acad. Sci.**USA*** 121**, e2404328121 (2024).Time-sharing Experiments for the Social Sciences.\n\n*TESS*[www.tessexperiments.org](http://www.tessexperiments.org)(2026).Coppock, A., Leeper, T. J. & Mullinix, K. J. Generalizability of heterogeneous treatment effect estimates across samples.\n\n*Proc. Natl Acad. Sci.**USA*** 115**, 12441–12446 (2018).Chang, S., Kennedy, A., Leonard, A. & List, J. A.\n\n*12 Best Practices for Leveraging Generative AI in Experimental Research*(National Bureau of Economic Research, 2024).Broska, D., Howes, M. & van Loon, A. The mixed subjects design: treating large language models as potentially informative observations.\n\n*Sociol. Methods Res.***54**, 1074–1109 (2025).Chu, J. Y. et al. Academics are more specific, and practitioners more sensitive, in forecasting interventions to strengthen democratic attitudes.\n\n*Proc. Natl Acad. Sci.**USA*** 121**, e2307008121 (2024).DellaVigna, S. & Linos, E. RCTs to scale: comprehensive evidence from two nudge units.\n\n*Econometrica***90**, 81–116 (2022).Dreber, A. et al. Using prediction markets to estimate the reproducibility of scientific research.\n\n*Proc. Natl Acad. Sci.**USA*** 112**, 15343–15347 (2015).Milkman, K. L. et al. Megastudies improve the impact of applied behavioural science.\n\n*Nature***600**, 478–483 (2021).Harding, J., D’Alessandro, W., Laskowski, N. G. & Long, R. AI language models cannot replace human research participants.\n\n*AI Soc.***39**, 2603–2605 (2024).Messeri, L. & Crockett, M. J. Artificial intelligence and illusions of understanding in scientific research.\n\n*Nature***627**, 49–58 (2024).Binz, M. et al. A foundation model to predict and capture human cognition.\n\n*Nature***644**, 1002–1009 (2025).Petrov, N. B., Serapio-García, G. & Rentfrow, J. Limited ability of LLMs to simulate human psychological behaviours: a psychometric analysis. Preprint at\n\n[https://arxiv.org/abs/2405.07248](https://arxiv.org/abs/2405.07248)(2024).Tjuatja, L., Chen, V., Wu, T., Talwalkwar, A. & Neubig, G. Do LLMs exhibit human-like response biases? A case study in survey design.\n\n*Trans. Assoc. Comput. Linguist.***12**, 1011–1026 (2024).Chen, Y., Hu, Y. & Lu, Y. Predicting field experiments with large language models. Preprint at\n\n[https://arxiv.org/abs/2504.01167](https://arxiv.org/abs/2504.01167)(2025).Wang, X. et al. Self-consistency improves chain of thought reasoning in language models. In\n\n*Proc. 11th International Conference on Learning Representations*(poster) (ICLR, 2023).Arora, S. et al. Ask me anything: a simple strategy for prompting language models. In\n\n*Proc. 11th International Conference on Learning Representations*(ICLR, 2023).Voronov, A., Wolf, L. & Ryabinin, M. Mind your format: towards consistent evaluation of in-context learning improvements. In\n\n*Findings of the Association for Computational Linguistics: ACL 2024*(eds Ku, L.-W. et al.) 6287–6310 (ACL, 2024).Hou, B., O’Connor, J., Andreas, J., Chang, S. & Zhang, Y. Promptboosting: black-box text classification with ten forward passes. In\n\n*Proc.**International Conference on Machine Learning*(eds Krause, A. et al.) 13309–13324 (PMLR, 2023).Open Science Collaboration. Estimating the reproducibility of psychological science.\n\n*Science***349**, aac4716 (2015).Milkman, K. L. et al. Megastudy shows that reminders boost vaccination but adding free rides does not.\n\n*Nature***631**, 179–188 (2024).Milkman, K. L. et al. A 680,000-person megastudy of nudges to encourage vaccination in pharmacies.\n\n*Proc. Natl Acad. Sci.**USA*** 119**, e2115126119 (2022).Zickfeld, J. H. et al. Effectiveness of ex ante honesty oaths in reducing dishonesty depends on content.\n\n*Nat. Hum. Behav.***9**, 169–187 (2025).Broockman, D. E., Kalla, J. L., Caballero, C. & Easton, M. Political practitioners poorly predict which messages persuade the public.\n\n*Proc. Natl Acad. Sci.**USA*** 121**, e2400076121 (2024).DellaVigna, S. & Vivalt, E. Forecasting social science: evidence from 100 projects.\n\n*National Bureau of Economic Research*[https://doi.org/10.3386/w34493](https://doi.org/10.3386/w34493)(2025).Allen, J., Watts, D. J. & Rand, D. G. Quantifying the impact of misinformation and vaccine-skeptical content on Facebook.\n\n*Science***384**, eadk3451 (2024).DellaVigna, S. & Pope, D. What motivates effort? Evidence and expert forecasts.\n\n*Rev. Econ. Stud.***85**, 1029–1069 (2018).Vlasceanu, M. et al. Addressing climate change with behavioral science: a global intervention tournament in 63 countries.\n\n*Sci. Adv.***10**, eadj5778 (2024).Tappin, B. M., Wittenberg, C., Hewitt, L. B., Berinsky, A. J. & Rand, D. G. Quantifying the potential persuasive returns to political microtargeting.\n\n*Proc. Natl Acad. Sci.**USA*** 120**, e2216261120 (2023).Voelkel, J. G. et al. Megastudy testing 25 treatments to reduce antidemocratic attitudes and partisan animosity.\n\n*Science***386**, eadh4764 (2024).Voelkel, J. G. et al. A registered report megastudy on the persuasiveness of the most-cited climate messages.\n\n*Nat. Clim. Chang.***16**, 214–225 (2026).Saccardo, S. et al. Field testing the transferability of behavioural science knowledge on promoting vaccinations.\n\n*Nat. Hum. Behav.***8**, 878–890 (2024).Mason, K. et al. A megastudy of behavioral interventions to increase voter registration ahead of the 2024 US presidential election. Preprint at PsyArXiv\n\n[https://doi.org/10.31234/osf.io/p6b2e_v3](https://doi.org/10.31234/osf.io/p6b2e_v3)(2025).Goldwert, D. et al. A megastudy of behavioral interventions to catalyze public, political, and financial climate advocacy.\n\n*PNAS Nexus***5**, pgaf400 (2026).Druckman, J. N.\n\n*Experimental Thinking*(Cambridge Univ. Press, 2022).Freese, J. & Peterson, D. Replication in social science.\n\n*Annu. Rev. Sociol.***43**, 147–165 (2017).Bai, X., Wang, A., Sucholutsky, I. & Griffiths, T. L. Explicitly unbiased large language models still form biased associations.\n\n*Proc. Natl Acad. Sci.**USA*** 122**, e2416228122 (2025).Wang, A., Morgenstern, J. & Dickerson, J. P. Large language models that replace human participants can harmfully misportray and flatten identity groups.\n\n*Nat. Mach. Intell.***7**, 400–411 (2025).Park, P. S., Schoenegger, P. & Zhu, C. Diminished diversity-of-thought in a standard large language model.\n\n*Behav. Res. Methods***56**, 5754–5770 (2024).Frank, M. C. Openly accessible LLMs can help us to understand human cognition.\n\n*Nat. Hum. Behav.***7**, 1825–1827 (2023).Wack, M., Ehrett, C., Linvill, D. & Warren, P. Generative propaganda: evidence of AI’s impact from a state-backed disinformation campaign.\n\n*PNAS Nexus***4**, pgaf083 (2025).Response retrieved from ChatGPT-4 on 22 May.\n\n*ChatGPT*[https://chatgpt.com/share/2a2b66d0-e9fd-421f-bedd-bb6f9354742a](https://chatgpt.com/share/2a2b66d0-e9fd-421f-bedd-bb6f9354742a)(2024).Almaatouq, A. et al. Beyond playing 20 questions with nature: integrative experiment design in the social and behavioral sciences.\n\n*Behav. Brain Sci.***47**, e33 (2024).Zhu, J.-Q., Xie, H., Arumugam, D., Wilson, R. C. & Griffiths, T. L. Using reinforcement learning to train large language models to explain human decisions. Preprint at\n\n[https://arxiv.org/abs/2505.11614](https://arxiv.org/abs/2505.11614)(2025).Tyner, A. H. et al. Investigating the replicability of the social and behavioural sciences.\n\n*Nature***652**, 143–150 (2026).Cremer, D. D. & Kasparov, G. AI should augment human intelligence, not replace it.\n\n*Harvard Business Review*[https://hbr.org/2021/03/ai-should-augment-human-intelligence-not-replace-it](https://hbr.org/2021/03/ai-should-augment-human-intelligence-not-replace-it)(2021).Cummins, J. The threat of analytic flexibility in using large language models to simulate human data: a call to attention. Preprint at\n\n[https://arxiv.org/abs/2509.13397](https://arxiv.org/abs/2509.13397)(2025).Obermeyer, Z., Powers, B., Vogeli, C. & Mullainathan, S. Dissecting racial bias in an algorithm used to manage the health of populations.\n\n*Science***366**, 447–453 (2019).Coppock, A. Generalizing from survey experiments conducted on mechanical turk: a replication approach.\n\n*Political Sci. Res. Methods***7**, 613–628 (2019).Mullinix, K. J., Leeper, T. J., Druckman, J. N. & Freese, J. The generalizability of survey experiments.\n\n*J. Exp. Political Sci.***2**, 109–138 (2015).Achiam, J. et al. GPT-4 technical report. Preprint at\n\n[https://arxiv.org/abs/2303.08774](https://arxiv.org/abs/2303.08774)(2023).Viechtbauer, W. metafor: Meta-Analysis Package for R v.4.8-0 (2015).\n\nRios, K., Roth, Z. C. & Coleman, T. J. III. The importance of scientists’ intellectual humility for communicating effectively across ideological and identity-based divides.\n\n*Proc. Natl Acad. Sci. USA***122**, e2400930121 (2025).\n\n## Acknowledgements\n\nWe thank N. Egan, K. Fuller, V. Isarraras, S. Kane, B. Kinder, A. Kissi, K. McPeek, S. Muldowney, C. Redekopp and T. Ren for research assistance. We are grateful to researchers who shared data, including J. Chu, S. DellaVigna, J. Druckman, A. Duckworth, D. Goldwert, N. Malhotra, K. Mason, K. Milkman, K. Okst, S. Saccardo, M. Vlasceanu and J. Voelkel. We also thank J. Allen, M. Atari, C. Bail, M. Bernstein, M. Dehghani, D. Broockman, K. Clayton, M. Crockett, D. Ganguli, D. Dillon, J. Eichstaedt, M. Frank, N. Goodman, D. Green, J. Kalla, P. Liang, J. McClelland, D. Rand, R. Orchinik, N. Otis, S. Rathje, B. Tappin, A. Van Loon, A. Waytz and D. Willner for invaluable feedback on the project.\n\n## Funding\n\nWe acknowledge funding from the Stanford Center on Philanthropy and Civil Society.\n\n## Author information\n\n### Authors and Affiliations\n\n### Contributions\n\nL.H., A.A. and R.W. designed the research. A.A., L.H., I.G. and R.W. assembled the experimental archives. L.H. and A.A. analysed the data. A.A., L.H. and R.W. wrote the paper. L.H. built the web tool.\n\n### Corresponding authors\n\n## Ethics declarations\n\n### Competing interests\n\nThe authors declare no competing interests.\n\n## Peer review\n\n### Peer review information\n\n*Nature* thanks Jamie Cummins, James Druckman, John Protzko and Youyou Wu for their contribution to the peer review of this work. [Peer reviewer reports](/articles/s41586-026-10742-x#MOESM3) are available.\n\n## Additional information\n\n**Publisher’s note** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.\n\n## Extended data figures and tables\n\n[Extended Data Fig. 1 Method.](/articles/s41586-026-10742-x/figures/4)\n\nWe re-analyze raw data from 70 nationally-representative U.S. studies and estimate treatment effects in a consistent manner. We use a Large Language Model to predict results of those same experiments, providing the original study materials and demographically-diverse participant profiles, and then calculate the average predicted response for each condition. We evaluate the accuracy of the model in terms of the correspondence between observed- and predicted- treatment effects.\n\n[Extended Data Fig. 2 Averaging across more LLM prompts improves predictive accuracy.](/articles/s41586-026-10742-x/figures/5)\n\nError bars indicate 95% confidence intervals for the mean Pearson correlation across 32 simulation runs. Each run uses a different random subset of prompts (drawn without replacement from 120). The CI reflects uncertainty in the mean across simulation runs, but does not include uncertainty over the sample of 70 experiments or the sample of 120 prompts.\n\n[Extended Data Fig. 3 The accuracy of LLM-derived predictions for demographic subgroups in the US.](/articles/s41586-026-10742-x/figures/6)\n\nError bars indicate 95% confidence intervals.\n\n## Supplementary information\n\n[Supplementary Information (download PDF )](https://static-content.springer.com/esm/art%3A10.1038%2Fs41586-026-10742-x/MediaObjects/41586_2026_10742_MOESM1_ESM.pdf)\n\nSupplementary Figs. 1–10, Tables 1–6 and text on the construction of the primary and secondary experimental archives, prompting strategy, layperson and expert forecasts, survey recruitment, analytical methods for both archives and the scientific applications, robustness checks and further applications and risk analyses. Supplementary Figs. 1–10 and Tables 1–6 provide additional detail on the study archives and methods, assess the robustness of LLM-based predictions, present further analyses of pilot-testing and intervention-selection use cases, and report detailed survey results from social scientists.\n\n## Rights and permissions\n\nSpringer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.\n\n## About this article\n\n### Cite this article\n\nAshokkumar, A., Hewitt, L., Ghezae, I. *et al.* Large language models can predict the results of social science experiments.\n*Nature* (2026). https://doi.org/10.1038/s41586-026-10742-x\n\nReceived:\n\nAccepted:\n\nPublished:\n\nVersion of record:\n\nDOI: https://doi.org/10.1038/s41586-026-10742-x", "url": "https://wpnews.pro/news/large-language-models-can-predict-the-results-of-social-science-experiments", "canonical_source": "https://www.nature.com/articles/s41586-026-10742-x", "published_at": "2026-07-08 18:53:03+00:00", "updated_at": "2026-07-08 19:12:30.848582+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "ai-research"], "entities": ["GPT-4", "Nature"], "alternates": {"html": "https://wpnews.pro/news/large-language-models-can-predict-the-results-of-social-science-experiments", "markdown": "https://wpnews.pro/news/large-language-models-can-predict-the-results-of-social-science-experiments.md", "text": "https://wpnews.pro/news/large-language-models-can-predict-the-results-of-social-science-experiments.txt", "jsonld": "https://wpnews.pro/news/large-language-models-can-predict-the-results-of-social-science-experiments.jsonld"}}