A Hybrid Theory and Data-driven Approach to Persuasion Detection with Large Language Models

Researchers from the University of Adelaide and the University of Western Australia developed a hybrid model using large language models (LLMs) to predict successful persuasion in online discourse, finding that epistemic emotion and willingness to share were the top predictors of belief change among eight features tested. The study, presented at the First Workshop on Integrating NLP and Psychology to Study Social Interactions (NLPSI) at ICWSM '25 in Copenhagen, Denmark, in June 2025, used LLM-generated ratings to build a random forest classification model, with applications in online influence detection and misinformation mitigation.

A Hybrid Theory and Data-driven Approach to Persuasion Detection with Large Language Models https://aclanthology.org/2025.nlpsi-1.5.pdf Gia Bao Hoang /people/gia-bao-hoang/ , Keith J Ransom /people/keith-j-ransom/ , Rachel Stephens /people/rachel-stephens/ , Carolyn Semmler /people/carolyn-semmler/ , Nicolas Fay /people/nicolas-fay-2800/ , Lewis Mitchell /people/lewis-mitchell/ Abstract Traditional psychological models of belief revision focus on face-to-face interactions, but with the rise of social media, more effective models are needed to capture belief revision at scale, in this rich text-based online discourse. Here, we use a hybrid approach, utilizing large language models LLMs to develop a model that predicts successful persuasion using features derived from psychological experiments.Our approach leverages LLM generated ratings of features previously examined in the literature to build a random forest classification model that predicts whether a message will result in belief change. Of the eight features tested, epistemic emotion and willingness to share to share were the top-ranking predictors of belief change in the model. Our findings provide insights into the characteristics of persuasive messages and demonstrate how LLMs can enhance models of successful persuasion based on psychological theory. Given these insights, this work has broader applications in fields such as online influence detection and misinformation mitigation, as well as measuring the effectiveness of online narratives. - Anthology ID: - 2025.nlpsi-1.5 - Volume: Proceedings of the First Workshop on Integrating NLP and Psychology to Study Social Interactions NLPSI @ICWSM ’25 /volumes/2025.nlpsi-1/ - Month: - June - Year: - 2025 - Address: - Copenhagen, Denmark - Editors: Aswathy Velutharambath /people/aswathy-velutharambath/unverified/ , Sofie Labat /people/sofie-labat/ , Neele Falk /people/neele-falk/ , Flor Miriam Plaza-del-Arco /people/flor-miriam-plaza-del-arco/ , Roman Klinger /people/roman-klinger/ , Véronique Hoste /people/veronique-hoste/unverified/ - Venues: NLPSI /venues/nlpsi/ | WS /venues/ws/ - SIG: - Publisher: - Association for the Advancement of Artificial Intelligence www.aaai.org - Note: - Pages: - 45–56 - Language: - URL: https://aclanthology.org/2025.nlpsi-1.5/ https://aclanthology.org/2025.nlpsi-1.5/ - DOI: 10.36190/2025.38 https://doi.org/10.36190/2025.38 - Cite ACL : - Gia Bao Hoang, Keith J Ransom, Rachel Stephens, Carolyn Semmler, Nicolas Fay, and Lewis Mitchell. 2025. A Hybrid Theory and Data-driven Approach to Persuasion Detection with Large Language Models https://aclanthology.org/2025.nlpsi-1.5/ . In Proceedings of the First Workshop on Integrating NLP and Psychology to Study Social Interactions NLPSI @ICWSM ’25 , pages 45–56, Copenhagen, Denmark. Association for the Advancement of Artificial Intelligence www.aaai.org . - Cite Informal : A Hybrid Theory and Data-driven Approach to Persuasion Detection with Large Language Models https://aclanthology.org/2025.nlpsi-1.5/ Hoang et al., NLPSI 2025 - PDF: https://aclanthology.org/2025.nlpsi-1.5.pdf https://aclanthology.org/2025.nlpsi-1.5.pdf