Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents

Researchers introduced Dialogue SWE-Bench, a benchmark for evaluating AI coding agents through dialogue with users, revealing that coding proficiency does not guarantee strong dialogue skills. The study proposes a schema-guided agent that improves dialogue capabilities by 3-14% over baselines, highlighting dialogue as a distinct performance dimension.

arXiv:2606.13995v1 Announce Type: new Abstract: AI coding agents have rapidly transformed software engineering, powering widely used interactive coding assistants. Despite their interactive real-world use, existing benchmarks evaluate them as fully-autonomous systems. In this work, we introduce Dialogue SWE-Bench, an automatic benchmark dataset for evaluating the ability of coding agents to resolve real-world software engineering problems through dialogue with a user. We design a novel, persona-grounded user simulator to support our task evaluation, and augment our task evaluation with automatic evaluations of dialogue quality. We also propose a new schema-guided agent, aimed at improving the dialogue capabilities of off-the-shelf coding agents, which improves over strong baselines by 3-14%. Our results indicate that better coding models do not always correspond to better dialogue models, suggesting that dialogue capability is a distinct and currently understudied dimension of coding agent performance.