TurnBench – a turn-taking benchmark for spoken conversation

Sesame AI Labs released TurnBench, a multi-domain benchmark for evaluating conversational turn-taking in spoken dialogue, featuring a 30-hour corpus of 154 dyadic dialogues from Mundo AI and ranking models by recall, false-positive rate, and latency. The benchmark's results show that no system is simultaneously fast, high-recall, and low on false positives, with VAP being the strongest model on both end-of-turn and interruption tracks, while humans begin turn transfers a median 151 ms before the turn ends.

TurnBench A multi-domain benchmark for evaluating conversational turn-taking. We hand-annotate end-of-turn and interruption events in dual-channel human conversations, and measure how accurately and how quickly models detect them. Leaderboard The ideal model is capable of high recall while maintaining a low false-positive rate and low latency. The following models are evaluated against our held-out test set and ranked by recall. - Recall: - share of true events detected higher is better . - FPR: - false-positive rate on test lower is better ; above 0.15 does not qualify. - Latency: - median delay after the event, p50 in ms lower is better . Click any model to open its full report: dev-set scores per conversation, with its predictions overlaid in the viewer. Dataset TurnBench evaluates models on a 30-hour corpus of studio-recorded, dual-channel dyadic speech. The corpus is provided by Mundo AI https://mundoai.world and features 154 dialogues, 106 actors, and is balanced across 6 conversation types 1. Every dialogue is labeled by three independent annotators Fleiss's κ = 0.78 where the ground truth label is derived from 2/3 consensus. Additionally, we include a training dataset, otoSpeech, hand-labeled under the same protocol as the evaluation set. All three splits are available on Hugging Face. - train - 104 h otoSpeech https://huggingface.co/datasets/otoearth/otoSpeech-full-duplex-turn-104h collected by Oto audio + annotations - dev - 7.3 h turn-benchmark-dev https://huggingface.co/datasets/mundo-ai/turn-benchmark-dev collected by Mundo AI audio + annotations - test - 22.9 h turn-benchmark-test https://huggingface.co/datasets/mundo-ai/turn-benchmark-test collected by Mundo AI audio only We created an interactive data viewer /conversations/42 for the public dev set. It is an audio player that displays each conversation with the annotated labels drawn on the waveform. You can compare these labels with a published baseline's predictions or submit your own. Evaluation Per conversation, TurnBench requires models to output a discrete list of times 2 for the following events: - End-of-turn - When a speaker has finished speaking and the floor is open for the other speaker to take.FP: declaring end-of-turn during a mid-turn pause. - Interruption - When a speaker takes the floor while the other is still talking.FP: declaring an interruption during a backchannel or because of noise, channel bleed, or echo. We open source the scorer, submission format, and baseline implementations on GitHub https://github.com/SesameAILabs/turnbench . Results We find end-of-turn recall is invariant to conversation type 1, while interruption false positives are not, tending to concentrate in casual conversations with high backchannel density. As expected, we find that acoustic detectors fire over-eagerly on silences, saturating recall but blowing past the false-positive budget. Semantic systems on the other hand are more selective but slower, staying in budget by waiting for more evidence. VAP is the strongest model on both the EOT and INT tracks, but no system is simultaneously fast, high-recall, and low on false positives. For reference, humans in the TurnBench corpus begin turn transfers a median 151 ms before the turn ends 3. We hope TurnBench is a valuable resource for the community to create more effective spoken interaction systems. Submissions We invite the community to improve upon these results, and provide the dev-set scorer /dev as a resource. Please email turnbench@sesame.com mailto:turnbench@sesame.com with your submission by including your predictions, organization name, and model name.