SAGE: State-Grounded, Abstention-Aware Evaluation of Task-Oriented Dialogue Agents

Researchers propose SAGE (State-Grounded Abstention-Aware Evaluation), a new framework for evaluating task-oriented dialogue agents that compiles workflow specifications and per-turn state diffs into atomic criteria, using a cascade of symbolic and encoder/NLI verifiers that abstain rather than guess. In tests across MultiWOZ, Schema-Guided Dialogue, and ABCD, SAGE-Core, its recommended operating point, decides 81–91% of criteria at zero paid LLM cost and matches or exceeds all LLM-as-a-judge baselines, including a state-aware GPT-4.1 judge that costs $4.7–8.0 per 1,000 turns versus SAGE-Core's $0. A two-annotator human audit (n=200, κ=0.94) confirms strong label fidelity on transcript-visible failure classes, though the authors note construct-validity limits from injected failures and partial symbolic circularity.

arXiv:2609.00434v1 Announce Type: new Abstract: Evaluating task-oriented dialogue agents requires judging not merely whether a reply reads well but whether each turn advances the underlying workflow state correctly--a distinction conventional holistic LLM judges can miss because they evaluate the available context as a single unit and require one or more full-model calls per turn. We propose SAGE State-Grounded Abstention-Aware Evaluation , which compiles a workflow specification and per-turn state diff into atomic, schema-grounded criteria and routes each through a cascade of symbolic and encoder/NLI verifiers that abstain rather than guess, aggregating criterion verdicts into a turn-level decision with an evidence trace. Its recommended operating point, SAGE-Core, decides 81--91% of criteria with only the compiler, symbolic rules, and on-device encoders--at zero paid LLM cost--while SAGE-LLM adds an optional focused-LLM fallback for open-class criteria. Across four slices spanning MultiWOZ, Schema-Guided Dialogue, and ABCD, no evaluated LLM-as-a-judge baseline--including a state-aware GPT-4.1 judge and cheaper GPT-4.1-mini variants--significantly exceeds SAGE-Core on any slice, even though the GPT-4.1 G-Eval judge costs $4.7--8.0 per 1,000 turns to SAGE-Core's $0. A two-annotator human audit n=200, $\kappa$=0.94 confirms strong label fidelity on the transcript-visible failure classes--where, excluding the weak-salience IUV class, SAGE-Core is statistically tied with the strongest LLM judge--and honestly scopes ignored-user-value as a state-consistency signal with weak broad-human salience. We analyze construct-validity limits from injected failures and partial symbolic circularity.