{"slug": "do-you-truly-love-me-benchmarking-llm-capability-on-hierarchical-pragmatic", "title": "“Do You Truly Love Me?” Benchmarking LLM Capability on Hierarchical Pragmatic Tactic Conversations", "summary": "A team led by Peter Q. Chen introduced HIPO, a benchmark published at AACL-IJCNLP 2026 that tests large language models on hierarchical pragmatic reasoning in multi-turn conversations, built from 4,088 benchmarking utterances and 6,350 contextual utterances across 1,131 dialogues drawn from 31 real-world-inspired scenarios. Benchmarking 22 state-of-the-art LLMs, the authors found models reached 83.3% accuracy on recognizing surface speech acts but only 32.2% on detecting veracity strategies and 45.1% on speaker intentions. The authors report that supervised fine-tuning on additional synthetic data generated by their pipeline improved performance on HIPO, suggesting a path for targeted capability improvement.", "body_md": "##### Abstract\n\nPragmatic reasoning, including inferring intent, manipulation, and hidden meaning in dialogue, is essential for trustworthy language understanding yet remains underexplored in current large language models (LLMs). We introduce\n**HIPO**, a new benchmark designed to assess\n\n**erarchical**\n\n__Hi__\n**ragmatic Tactic in multi-turn C**\n\n__P__\n**nversations. Inspired by linguistic pragmatics, HIPO operationalizes a hierarchical annotation schema for multi-turn conversations and labels each utterance along one dialogue-level goal and three utterance-level dimensions: communicative intention (**\n\n__o__\n*why*), veracity strategy (\n\n*how*), and illocutionary act (\n\n*what*). To ensure reliable supervision, we design a structured generation pipeline to generate high-quality synthetic dialogues. The benchmark comprises over 4,088 benchmarking utterances with 6,350 contextual utterances across 1,131 dialogues drawn from 31 real-world-inspired scenarios. Benchmarking 22 state-of-the-art LLMs reveals a striking gap: while models achieve higher performance at recognizing surface speech acts(83.3% accuracy), they perform poorly on detecting veracity strategies(32.2%) and speaker intentions(45.1%). These results suggest current LLMs remain challenged by the pragmatic inferences operationalized in HIPO, particularly those involving information manipulation and intended effects beyond literal content. We further show that supervised fine-tuning on additional synthetic data generated by our pipeline can improve performance on HIPO, suggesting a practical path for targeted capability improvement within this benchmark setting.\n\n- Anthology ID:\n- 2026.aacl-long.191\n- Volume:\n- [Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)](https://aclanthology.org/volumes/2026.aacl-long/)\n- Month:\n- November\n- Year:\n- 2026\n- Address:\n- Hengqin, China\n- Editors:\n- [Yue Zhang](https://aclanthology.org/people/yue-zhang/unverified/) ,[Ruifeng Xu](https://aclanthology.org/people/ruifeng-xu/) ,[Ayu Purwarianti](https://aclanthology.org/people/ayu-purwarianti/) ,[Tong Rong](https://aclanthology.org/people/tong-rong/unverified/) ,[Jun Suzuki](https://aclanthology.org/people/jun-suzuki/)\n- Venues:\n- [AACL](https://aclanthology.org/venues/aacl/) |[IJCNLP](https://aclanthology.org/venues/ijcnlp/)\n- SIG:\n- Publisher:\n- Association for Computational Linguistics\n- Note:\n- Pages:\n- 4261–4281\n- Language:\n- URL:\n- [https://aclanthology.org/2026.aacl-long.191/](https://aclanthology.org/2026.aacl-long.191/)\n- DOI:\n- Cite (ACL):\n- Peter Q. Chen, Pangjing Wu, Zhuohang Jiang, Xiaodong Li, and Peter H. F. Ng. 2026. [“Do You Truly Love Me?” Benchmarking LLM Capability on Hierarchical Pragmatic Tactic Conversations](https://aclanthology.org/2026.aacl-long.191/) . In*Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)* , pages 4261–4281, Hengqin, China. Association for Computational Linguistics.\n- Cite (Informal):\n- [“Do You Truly Love Me?” Benchmarking LLM Capability on Hierarchical Pragmatic Tactic Conversations](https://aclanthology.org/2026.aacl-long.191/) (Chen et al., AACL-IJCNLP 2026)\n- PDF:\n- [https://aclanthology.org/2026.aacl-long.191.pdf](https://aclanthology.org/2026.aacl-long.191.pdf)", "url": "https://wpnews.pro/news/do-you-truly-love-me-benchmarking-llm-capability-on-hierarchical-pragmatic", "canonical_source": "https://aclanthology.org/2026.aacl-long.191/", "published_at": "2026-10-31 00:00:00+00:00", "updated_at": "2026-11-05 02:20:10.369325+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "natural-language-processing", "artificial-intelligence"], "entities": ["HIPO", "Peter Q. Chen", "Pangjing Wu", "Zhuohang Jiang", "Xiaodong Li", "Peter H. F. Ng", "AACL-IJCNLP 2026", "Association for Computational Linguistics"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/do-you-truly-love-me-benchmarking-llm-capability-on-hierarchical-pragmatic", "markdown": "https://wpnews.pro/news/do-you-truly-love-me-benchmarking-llm-capability-on-hierarchical-pragmatic.md", "text": "https://wpnews.pro/news/do-you-truly-love-me-benchmarking-llm-capability-on-hierarchical-pragmatic.txt", "jsonld": "https://wpnews.pro/news/do-you-truly-love-me-benchmarking-llm-capability-on-hierarchical-pragmatic.jsonld"}}