A multi-domain benchmark for evaluating conversational turn-taking. We hand-annotate end-of-turn and interruption events in dual-channel human conversations, and measure how accurately and how quickly models detect them.
Leaderboard #
The ideal model is capable of high recall while maintaining a low false-positive rate and low latency. The following models are evaluated against our held-out test set and ranked by recall.
- Recall:
- share of true events detected (higher is better).
- FPR:
- false-positive rate on test (lower is better); above 0.15 does not qualify.
- Latency:
- median delay after the event, p50 in ms (lower is better).
Click any model to open its full report: dev-set scores per conversation, with its predictions overlaid in the viewer.
Dataset #
TurnBench evaluates models on a 30-hour corpus of studio-recorded, dual-channel dyadic speech. The corpus is provided by Mundo AI and features 154 dialogues, 106 actors, and is balanced across 6 conversation types 1. Every dialogue is labeled by three independent annotators (Fleiss's κ = 0.78) where the ground truth label is derived from 2/3 consensus.
Additionally, we include a training dataset, otoSpeech, hand-labeled under the same protocol as the evaluation set. All three splits are available on Hugging Face.
- train
- 104 h
otoSpeech collected by Oto
audio + annotations - dev
- 7.3 h
turn-benchmark-dev collected by Mundo AI
audio + annotations - test
- 22.9 h
turn-benchmark-test collected by Mundo AI
audio only
We created an interactive data viewer for the public dev set. It is an audio player that displays each conversation with the annotated labels drawn on the waveform. You can compare these labels with a published baseline's predictions or submit your own.
Evaluation #
Per conversation, TurnBench requires models to output a discrete list of times 2 for the following events:
- End-of-turn
- When a speaker has finished speaking and the floor is open for the other speaker to take.FP: declaring end-of-turn during a mid-turn .
- Interruption
- When a speaker takes the floor while the other is still talking.FP: declaring an interruption during a backchannel or because of noise, channel bleed, or echo.
We open source the scorer, submission format, and baseline implementations on GitHub.
Results #
We find end-of-turn recall is invariant to conversation type 1, while interruption false positives are not, tending to concentrate in casual conversations with high backchannel density. As expected, we find that acoustic detectors fire over-eagerly on silences, saturating recall but blowing past the false-positive budget. Semantic systems on the other hand are more selective but slower, staying in budget by waiting for more evidence.
VAP is the strongest model on both the EOT and INT tracks, but no system is simultaneously fast, high-recall, and low on false positives. For reference, humans in the TurnBench corpus begin turn transfers a median 151 ms before the turn ends 3. We hope TurnBench is a valuable resource for the community to create more effective spoken interaction systems.
Submissions #
We invite the community to improve upon these results, and provide the dev-set scorer as a resource. Please email turnbench@sesame.com with your submission by including your predictions, organization name, and model name.