{"slug": "sage-state-grounded-abstention-aware-evaluation-of-task-oriented-dialogue-agents", "title": "SAGE: State-Grounded, Abstention-Aware Evaluation of Task-Oriented Dialogue Agents", "summary": "Researchers propose SAGE (State-Grounded Abstention-Aware Evaluation), a new framework for evaluating task-oriented dialogue agents that compiles workflow specifications and per-turn state diffs into atomic criteria, using a cascade of symbolic and encoder/NLI verifiers that abstain rather than guess. In tests across MultiWOZ, Schema-Guided Dialogue, and ABCD, SAGE-Core, its recommended operating point, decides 81–91% of criteria at zero paid LLM cost and matches or exceeds all LLM-as-a-judge baselines, including a state-aware GPT-4.1 judge that costs $4.7–8.0 per 1,000 turns versus SAGE-Core's $0. A two-annotator human audit (n=200, κ=0.94) confirms strong label fidelity on transcript-visible failure classes, though the authors note construct-validity limits from injected failures and partial symbolic circularity.", "body_md": "arXiv:2609.00434v1 Announce Type: new\nAbstract: Evaluating task-oriented dialogue agents requires judging not merely whether a reply reads well but whether each turn advances the underlying workflow state correctly--a distinction conventional holistic LLM judges can miss because they evaluate the available context as a single unit and require one or more full-model calls per turn. We propose SAGE (State-Grounded Abstention-Aware Evaluation), which compiles a workflow specification and per-turn state diff into atomic, schema-grounded criteria and routes each through a cascade of symbolic and encoder/NLI verifiers that abstain rather than guess, aggregating criterion verdicts into a turn-level decision with an evidence trace. Its recommended operating point, SAGE-Core, decides 81--91% of criteria with only the compiler, symbolic rules, and on-device encoders--at zero paid LLM cost--while SAGE-LLM adds an optional focused-LLM fallback for open-class criteria. Across four slices spanning MultiWOZ, Schema-Guided Dialogue, and ABCD, no evaluated LLM-as-a-judge baseline--including a state-aware GPT-4.1 judge and cheaper GPT-4.1-mini variants--significantly exceeds SAGE-Core on any slice, even though the GPT-4.1 G-Eval judge costs $4.7--8.0 per 1,000 turns to SAGE-Core's $0. A two-annotator human audit (n=200, $\\kappa$=0.94) confirms strong label fidelity on the transcript-visible failure classes--where, excluding the weak-salience IUV class, SAGE-Core is statistically tied with the strongest LLM judge--and honestly scopes ignored-user-value as a state-consistency signal with weak broad-human salience. We analyze construct-validity limits from injected failures and partial symbolic circularity.", "url": "https://wpnews.pro/news/sage-state-grounded-abstention-aware-evaluation-of-task-oriented-dialogue-agents", "canonical_source": "https://www.machinebrief.com/news/sage-state-grounded-abstention-aware-evaluation-of-task-orie-8cnl", "published_at": "2026-09-02 04:00:00+00:00", "updated_at": "2026-09-02 04:24:52.820859+00:00", "lang": "en", "topics": ["artificial-intelligence", "natural-language-processing", "ai-research", "ai-tools"], "entities": ["SAGE", "SAGE-Core", "SAGE-LLM", "GPT-4.1", "MultiWOZ", "Schema-Guided Dialogue", "ABCD"], "alternates": {"html": "https://wpnews.pro/news/sage-state-grounded-abstention-aware-evaluation-of-task-oriented-dialogue-agents", "markdown": "https://wpnews.pro/news/sage-state-grounded-abstention-aware-evaluation-of-task-oriented-dialogue-agents.md", "text": "https://wpnews.pro/news/sage-state-grounded-abstention-aware-evaluation-of-task-oriented-dialogue-agents.txt", "jsonld": "https://wpnews.pro/news/sage-state-grounded-abstention-aware-evaluation-of-task-oriented-dialogue-agents.jsonld"}}