Gemini, ElevenLabs top new Voice AI benchmark

Hume released a new voice AI benchmark, Real World VoiceEQ, evaluating over 40 models across 15 dimensions and 60 metrics based on 700,000 human judgments. Google Gemini led text-to-speech and speech understanding, OpenAI's GPT Realtime Mini led speech-to-speech, and ElevenLabs led automatic speech recognition. The benchmark aims to identify model strengths for specific use cases rather than a single best model.

One of AI's big promises is making your workflows faster, and talking to your computer is one of the big unlocks. Voice AI models are booming because they let people interact with AI more seamlessly. As a result, minor latency, word errors, and misinterpretation of emotional cues, speech, or accents can greatly hinder the experience. Yet, Hume found that traditional benchmarks only accounted for technicalities rather than actual human interaction, so it built a new benchmark, "Real World VoiceEQ," to evaluate these models' performance more accurately. "Our team has spent more than a decade researching human expression and emotional intelligence, and Hume was founded to make voice AI more emotionally intelligent," Andrew Ettinger, Hume's CEO, told The Deep View. "We saw this benchmark as a way to share what we’ve learned and contribute to the field’s progress." The benchmark goes beyond technical correctness, also evaluating the models' ability to understand emotional cues, communicate naturally, and be reliable in real-world conversations. It does so by evaluating more than 40 voice models across more than 15 key evaluation dimensions and more than 60 metrics, and is grounded in 700,000 human judgments, making it the largest human evaluation of voice AI to date, according to the release. Every evaluation is also conducted using Kairos, Hume's voice-native evaluation platform. Contrary to what you may think, the purpose of the benchmarks isn't to find the "best" voice model, but rather to identify the strengths and weaknesses that make certain models better suited to different needs. Here is where the top models stand in the benchmark: Text-to-Speech : Google Gemini achieved the strongest performance Speech-to-Speech : OpenAI's GPT Realtime Mini achieved the strongest performance Automatic Speech Recognition : ElevenLabs emerged as the strongest overall ASR system Speech Understanding : Google Gemini led the Speech Understanding category overall "Our results show there isn’t one 'best' voice model. Different systems have different strengths and trade-offs," said Ettinger. "A good leaderboard helps companies choose based on what they actually need, whether that’s emotional intelligence, precise transcription, expressive speech, or reliability in noisy environments." Our Deeper View Hume found that current models are being fine-tuned to pass and perform well on popular benchmarks, without those capabilities necessarily translating to better performance in real-world use cases. This is a major issue that impacts other AI applications as well. For instance, at launch, many models achieve high benchmark performance in certain physics applications. Yet the models will fail to answer simple questions accurately, highlighting that benchmarks don't always translate to everyday use cases. More companies should adopt an approach of creating benchmarks that are fine-tuned to practical needs, rather than just sounding impressive based on pure data. The trap is that those impressive stats are what get power users, enterprises, and investors to buy in.