Abstract
Pragmatic reasoning, including inferring intent, manipulation, and hidden meaning in dialogue, is essential for trustworthy language understanding yet remains underexplored in current large language models (LLMs). We introduce HIPO, a new benchmark designed to assess
erarchical
Hi ragmatic Tactic in multi-turn C
P nversations. Inspired by linguistic pragmatics, HIPO operationalizes a hierarchical annotation schema for multi-turn conversations and labels each utterance along one dialogue-level goal and three utterance-level dimensions: communicative intention (
o
*why*), veracity strategy (
*how*), and illocutionary act (
what). To ensure reliable supervision, we design a structured generation pipeline to generate high-quality synthetic dialogues. The benchmark comprises over 4,088 benchmarking utterances with 6,350 contextual utterances across 1,131 dialogues drawn from 31 real-world-inspired scenarios. Benchmarking 22 state-of-the-art LLMs reveals a striking gap: while models achieve higher performance at recognizing surface speech acts(83.3% accuracy), they perform poorly on detecting veracity strategies(32.2%) and speaker intentions(45.1%). These results suggest current LLMs remain challenged by the pragmatic inferences operationalized in HIPO, particularly those involving information manipulation and intended effects beyond literal content. We further show that supervised fine-tuning on additional synthetic data generated by our pipeline can improve performance on HIPO, suggesting a practical path for targeted capability improvement within this benchmark setting.
- Anthology ID:
- 2026.aacl-long.191
- Volume:
- Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
- Month:
- November
- Year:
- 2026
- Address:
- Hengqin, China
- Editors:
- Yue Zhang ,Ruifeng Xu ,Ayu Purwarianti ,Tong Rong ,Jun Suzuki
- Venues:
- [AACL](https://aclanthology.org/venues/aacl/) |[IJCNLP](https://aclanthology.org/venues/ijcnlp/)
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 4261–4281
- Language:
- URL:
- [https://aclanthology.org/2026.aacl-long.191/](https://aclanthology.org/2026.aacl-long.191/)
- DOI:
- Cite (ACL):
- Peter Q. Chen, Pangjing Wu, Zhuohang Jiang, Xiaodong Li, and Peter H. F. Ng. 2026. “Do You Truly Love Me?” Benchmarking LLM Capability on Hierarchical Pragmatic Tactic Conversations . InProceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 4261–4281, Hengqin, China. Association for Computational Linguistics.
- Cite (Informal):
- [“Do You Truly Love Me?” Benchmarking LLM Capability on Hierarchical Pragmatic Tactic Conversations](https://aclanthology.org/2026.aacl-long.191/) (Chen et al., AACL-IJCNLP 2026)
- PDF:
- [https://aclanthology.org/2026.aacl-long.191.pdf](https://aclanthology.org/2026.aacl-long.191.pdf)