MADS: Multi-Agent Dialog Simulation for Robust Conversational Agents

Researchers introduced MADS (Multi-Agent Dialog Simulation), a synthetic data generation pipeline that injects adversarial conversation into task-oriented dialogue, and used it to build MADS-Bench, a benchmark spanning six domains, 55 personas, and 1,095 conversations with roughly 66% containing failure-prone scenarios. The team evaluated model robustness with a two-stage training approach combining supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR), addressing what they describe as an underrepresentation of adversarial conversation trajectories in existing function-calling benchmarks.

arXiv:2610.12757v1 Announce Type: new Abstract: Task-oriented conversational agents remain fragile under realistic conditions, especially when users exhibit non-cooperative behavior. Existing function-calling benchmarks underrepresent adversarial conversation trajectories, thereby limiting the training resources available for developing robust agents. We present MADS Multi-Agent Dialog Simulation , a synthetic data generation pipeline that produces structured, diverse dialogue trajectories. We inject adversarial conversation, producing validated conversation trajectories that capture realistic, unhappy paths. Using MADS, we construct MADS-Bench: a benchmark spanning six domains, 55 personas, and 1095 conversations, with approximately 66% of conversations containing failure-prone conversation scenarios. We evaluate the robustness of the trained models using a two-stage training approach that combines supervised fine-tuning SFT and reinforcement learning with verifiable rewards RLVR .