“Do You Truly Love Me?” Benchmarking LLM Capability on Hierarchical Pragmatic Tactic Conversations

A team led by Peter Q. Chen introduced HIPO, a benchmark published at AACL-IJCNLP 2026 that tests large language models on hierarchical pragmatic reasoning in multi-turn conversations, built from 4,088 benchmarking utterances and 6,350 contextual utterances across 1,131 dialogues drawn from 31 real-world-inspired scenarios. Benchmarking 22 state-of-the-art LLMs, the authors found models reached 83.3% accuracy on recognizing surface speech acts but only 32.2% on detecting veracity strategies and 45.1% on speaker intentions. The authors report that supervised fine-tuning on additional synthetic data generated by their pipeline improved performance on HIPO, suggesting a path for targeted capability improvement.

Abstract Pragmatic reasoning, including inferring intent, manipulation, and hidden meaning in dialogue, is essential for trustworthy language understanding yet remains underexplored in current large language models LLMs . We introduce HIPO , a new benchmark designed to assess erarchical Hi ragmatic Tactic in multi-turn C P nversations. Inspired by linguistic pragmatics, HIPO operationalizes a hierarchical annotation schema for multi-turn conversations and labels each utterance along one dialogue-level goal and three utterance-level dimensions: communicative intention o why , veracity strategy how , and illocutionary act what . To ensure reliable supervision, we design a structured generation pipeline to generate high-quality synthetic dialogues. The benchmark comprises over 4,088 benchmarking utterances with 6,350 contextual utterances across 1,131 dialogues drawn from 31 real-world-inspired scenarios. Benchmarking 22 state-of-the-art LLMs reveals a striking gap: while models achieve higher performance at recognizing surface speech acts 83.3% accuracy , they perform poorly on detecting veracity strategies 32.2% and speaker intentions 45.1% . These results suggest current LLMs remain challenged by the pragmatic inferences operationalized in HIPO, particularly those involving information manipulation and intended effects beyond literal content. We further show that supervised fine-tuning on additional synthetic data generated by our pipeline can improve performance on HIPO, suggesting a practical path for targeted capability improvement within this benchmark setting. - Anthology ID: - 2026.aacl-long.191 - Volume: - Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics Volume 1: Long Papers https://aclanthology.org/volumes/2026.aacl-long/ - Month: - November - Year: - 2026 - Address: - Hengqin, China - Editors: - Yue Zhang https://aclanthology.org/people/yue-zhang/unverified/ , Ruifeng Xu https://aclanthology.org/people/ruifeng-xu/ , Ayu Purwarianti https://aclanthology.org/people/ayu-purwarianti/ , Tong Rong https://aclanthology.org/people/tong-rong/unverified/ , Jun Suzuki https://aclanthology.org/people/jun-suzuki/ - Venues: - AACL https://aclanthology.org/venues/aacl/ | IJCNLP https://aclanthology.org/venues/ijcnlp/ - SIG: - Publisher: - Association for Computational Linguistics - Note: - Pages: - 4261–4281 - Language: - URL: - https://aclanthology.org/2026.aacl-long.191/ https://aclanthology.org/2026.aacl-long.191/ - DOI: - Cite ACL : - Peter Q. Chen, Pangjing Wu, Zhuohang Jiang, Xiaodong Li, and Peter H. F. Ng. 2026. “Do You Truly Love Me?” Benchmarking LLM Capability on Hierarchical Pragmatic Tactic Conversations https://aclanthology.org/2026.aacl-long.191/ . In Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics Volume 1: Long Papers , pages 4261–4281, Hengqin, China. Association for Computational Linguistics. - Cite Informal : - “Do You Truly Love Me?” Benchmarking LLM Capability on Hierarchical Pragmatic Tactic Conversations https://aclanthology.org/2026.aacl-long.191/ Chen et al., AACL-IJCNLP 2026 - PDF: - https://aclanthology.org/2026.aacl-long.191.pdf https://aclanthology.org/2026.aacl-long.191.pdf