Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment

A study of 49 programmers interacting with GitHub Copilot to assess 148 HIPAA-derived Non-Functional Requirements against the iTrust codebase found that developers tend to agree with LLM assessments, but accuracy against expert ground truth is low. Longer system responses and more information-providing turns negatively affect user satisfaction, whereas proactive interactions positively affect it, according to researchers from the University of Notre Dame and other institutions presenting at the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue.

Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment https://aclanthology.org/2026.sigdial-1.7.pdf Ali Pourghasemi Fatideh /people/ali-pourghasemi-fatideh/unverified/ , Wilder Baldwin /people/wilder-baldwin/unverified/ , Maria Dhakal /people/maria-dhakal/unverified/ , Collin McMillan /people/collin-mcmillan/ , Sepideh Ghanavati /people/sepideh-ghanavati/ Abstract LLM-based dialogue assistants have become mainstream tools for software developers, yet current evaluation benchmarks focus exclusively on functional correctness. This leaves a critical gap in assessing the quality and accuracy of these conversations when handling Non-Functional Requirements NFRs , which are inherently vague, context-dependent, and involve many parts of a program. Evaluating how well these systems support collaborative reasoning about NFRs requires methods that go beyond single-turn accuracy to capture both the correctness of the system’s outputs and the quality of the multi-turn interaction. In this paper, we investigate the accuracy and quality of multi-turn conversations between developers and an LLM-based agent in the domain of Health Insurance Portability and Accountability Act HIPAA regulatory compliance, a representative case of regulatory NFRs. We hired 49 programmers to interact with GitHub Copilot to assess 148 HIPAA-derived NFRs against the iTrust codebase, a system designed to comply with HIPAA regulations, across three dimensions: requirement satisfaction level, reasoning, and code localization. We find that developers tend to agree with LLM assessments, but accuracy against expert ground truth is low. We model user satisfaction and find that longer system responses and more information-providing turns negatively affect user satisfaction, whereas proactive interactions positively affect it. Our findings provide insights for designing LLM-based dialogue systems that support NFR assessment.- Anthology ID: - 2026.sigdial-1.7 - Volume: Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue /volumes/2026.sigdial-1/ - Month: - August - Year: - 2026 - Address: - Atlanta, Georgia, USA - Editors: Jinho D. Choi /people/jinho-d-choi/ , Yun-Nung Chen /people/yun-nung-chen/ , Kotaro Funakoshi /people/kotaro-funakoshi/ , Ali Emami /people/ali-emami/ - Venue: SIGDIAL /venues/sigdial/ - SIG: SIGDIAL /sigs/sigdial/ - Publisher: - Association for Computational Linguistics - Note: - Pages: - 91–106 - Language: - URL: https://aclanthology.org/2026.sigdial-1.7/ https://aclanthology.org/2026.sigdial-1.7/ - DOI: - Cite ACL : - Ali Pourghasemi Fatideh, Wilder Baldwin, Maria Dhakal, Collin McMillan, and Sepideh Ghanavati. 2026. Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment https://aclanthology.org/2026.sigdial-1.7/ . In Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue , pages 91–106, Atlanta, Georgia, USA. Association for Computational Linguistics. - Cite Informal : Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment https://aclanthology.org/2026.sigdial-1.7/ Pourghasemi Fatideh et al., SIGDIAL 2026 - PDF: https://aclanthology.org/2026.sigdial-1.7.pdf https://aclanthology.org/2026.sigdial-1.7.pdf