The Utility of LLM Text Generation in Longitudinal Psychological Datasets

Researchers Jari Zegers and Bennett Kleinberg found limited evidence that large language model (LLM) text generation can predict psychological variables in longitudinal datasets, but observed topic differences between generated and ground-truth texts. In a study presented at the First Workshop on Integrating NLP and Psychology to Study Social Interactions (NLPSI) @ICWSM ’25 in Copenhagen, Denmark, in June 2025, they prompted an LLM with three waves of texts from a longitudinal panel dataset to generate a wave 4 text, then compared generated texts to ground truth using cosine similarity on embeddings. The study, published by the Association for the Advancement of Artificial Intelligence (AAAI) on pages 92–94, found no significant association between text similarity and psychological variables, leaving the explanation for text differences under investigation.

Abstract As part of this ongoing work, we prompted an LLM with three waves of texts from a longitudinal panel dataset to generate a text for wave 4. We compared generated to ground truth texts using cosine similarity on embeddings and tested whether text similarity was associated with psychological variables. We found limited evidence for an association but do find differences in the topics used in generated versus ground-truth texts. An explanation for differences in text similarities remains the subject of ongoing investigation.- Anthology ID: - 2025.nlpsi-1.10 - Volume: Proceedings of the First Workshop on Integrating NLP and Psychology to Study Social Interactions NLPSI @ICWSM ’25 /volumes/2025.nlpsi-1/ - Month: - June - Year: - 2025 - Address: - Copenhagen, Denmark - Editors: Aswathy Velutharambath /people/aswathy-velutharambath/unverified/ , Sofie Labat /people/sofie-labat/ , Neele Falk /people/neele-falk/ , Flor Miriam Plaza-del-Arco /people/flor-miriam-plaza-del-arco/ , Roman Klinger /people/roman-klinger/ , Véronique Hoste /people/veronique-hoste/unverified/ - Venues: NLPSI /venues/nlpsi/ | WS /venues/ws/ - SIG: - Publisher: - Association for the Advancement of Artificial Intelligence www.aaai.org - Note: - Pages: - 92–94 - Language: - URL: https://aclanthology.org/2025.nlpsi-1.10/ https://aclanthology.org/2025.nlpsi-1.10/ - DOI: 10.36190/2025.39 https://doi.org/10.36190/2025.39 - Cite ACL : - Jari Zegers and Bennett Kleinberg. 2025. The Utility of LLM Text Generation in Longitudinal Psychological Datasets https://aclanthology.org/2025.nlpsi-1.10/ . In Proceedings of the First Workshop on Integrating NLP and Psychology to Study Social Interactions NLPSI @ICWSM ’25 , pages 92–94, Copenhagen, Denmark. Association for the Advancement of Artificial Intelligence www.aaai.org . - Cite Informal : The Utility of LLM Text Generation in Longitudinal Psychological Datasets https://aclanthology.org/2025.nlpsi-1.10/ Zegers & Kleinberg, NLPSI 2025 - PDF: https://aclanthology.org/2025.nlpsi-1.10.pdf https://aclanthology.org/2025.nlpsi-1.10.pdf