# New STT Models in 2026: Why Benchmarks Are Not Enough for Voice AI

> Source: <https://www.famulor.io/blog/stt-model-benchmarks-voice-ai-production-tests-2026>
> Published: 2026-09-06 16:27:00+00:00

### Summarize Content With:

Two major speech-to-text launches arrived within days of each other:
Meta introduced Muse Voice Transcribe on September 1, 2026, and
Microsoft followed with MAI-Transcribe-2 on September 3. Both deserve
the attention of enterprise voice AI teams. Neither a leaderboard
position nor a vendor-reported average word error rate, however, can
tell you how a model will perform on **your** callers
across **your** telephony path.

The practical answer is to use public STT benchmarks for Voice AI as
a screening tool, then require a held-out, production-like phone-audio
evaluation before switching. This article examines Muse and MAI as
current market developments. It is **not an announcement that
either model is available in Famulor**.

**Key Takeaways**

- Meta and Microsoft document capabilities that matter to live conversations, including streaming, diarization, code-switching and contextual biasing.
- Their published scores are vendor-reported results within specific benchmark conditions, not forecasts for an enterprise phone workload.
- Recent research shows that strong public ASR scores can sometimes hide benchmark-conditioned behavior.
- A defensible evaluation measures business-critical meaning, streaming behavior and real phone conditions by language and scenario.
- The Famulor sources reviewed for this article do not currently document support for Muse Voice Transcribe or MAI-Transcribe-2.

## What changed in speech recognition this week?

The new releases illustrate a broader direction in speech recognition: from producing a final transcript toward interpreting a live audio stream with speaker attribution, endpointing, language switching and contextual controls. Yet the public information does not support naming a winner for enterprise phone calls.

| Dimension | Meta Muse Voice Transcribe | Microsoft MAI-Transcribe-2 | 
|---|---|---|
| Release date | September 1, 2026 | September 3, 2026 | 
| Documented focus | Real-time audio perception combining streaming ASR, diarization and endpointing | Transcription with diarization, word-level timestamps, language identification and transcript styles | 
| Language statement | Meta says it trained the model on more than 70 languages and extensively verified 25 for the initial release | Microsoft reports FLEURS results across 60 languages; its current support table lists German and additional language codes | 
| Adaptation controls | Language, keyword and context biasing | Keyword biasing plus `verbatim` and`clean` transcript styles | 
| Stated access paths | Meta Model API, Meta AI for Mac and Muse Code | Microsoft Foundry, MAI Playground and OpenRouter; Microsoft Learn also documents Fast Transcription and Voice Live input transcription | 
| Benchmark claim | Meta reported first place on the Artificial Analysis streaming STT ranking and public diarization benchmarks as of September 1 | Microsoft reported 5.2% average WER on FLEURS across 60 languages and second place on the Artificial Analysis WER leaderboard | 

Meta’s [official
Muse announcement](https://research.meta.ai/blog/introducing-muse-voice-transcribe) says the architecture processes audio in 80 ms
chunks and uses adaptive delay to balance accuracy against time to a
final transcript. That is an architectural detail, not a claim of 80 ms
end-to-end call latency. Meta also says Muse can process audio longer
than an hour and more than 20 speakers without required post-processing.
These are vendor specifications, not independently reproduced telephony
results in this article.

Microsoft’s [MAI-Transcribe-2
launch post](https://microsoft.ai/news/mai-transcribe-2-is-the-fastest-most-accurate-and-cheapest-speech-recognition-model-in-the-world/) lists code-switching, robustness to noisy audio and
automatic language identification among the model’s capabilities. The
corresponding [Microsoft
Learn documentation](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/mai-transcribe) describes MAI as an input-transcription option
in Voice Live. This establishes a documented real-time route in
Microsoft’s stack, but it does not establish a native integration in
other platforms.

For a model-focused view of other current options, see Famulor’s
guide to [Gemini
3.5, Soniox v5 and Linden-1](https://www.famulor.io/blog/gemini-35-vs-soniox-v5-vs-linden-1-stt-guide-2026). The decision addressed here is
deliberately narrower: how much weight a buyer should give benchmark
claims when the production workload is phone audio.

## Why the benchmark warning is especially timely

A [Hugging
Face research article by Hume AI authors](https://huggingface.co/blog/asr-benchmark-optimization), published on August 21,
2026, asks whether ASR systems always transcribe the acoustic evidence
in public test sets or can sometimes reproduce familiar reference text.
Its companion [preprint by
Lebryk and colleagues](https://arxiv.org/abs/2608.19936), submitted on August 20, evaluated 11 widely
used open-source ASR models with three probe families:

1. **Reference disagreement:** the audio conflicts with
the benchmark transcript.
2. **Masked entities and numbers:** meaningful parts of
the audio are removed or obscured.
3. **Orthographic switching:** the test checks behavior
where several spellings are plausible.

The researchers found that high-performing systems sometimes produced benchmark reference wording even when the audio contradicted it or no longer contained the relevant material. In the reported LibriSpeech masked-number experiment, some strong benchmark performers recovered removed numbers in roughly 30–40% of examples. The effect generally weakened on freshly collected data.

The scope boundary matters: this work evaluated **neither Muse
Voice Transcribe nor MAI-Transcribe-2**. It does not rebut either
company’s launch claims. Instead, it strengthens the case for a better
evaluation design. Our editorial inference is that a public leaderboard
is useful for identifying candidates, while a production decision should
depend on held-out audio and references that a model could not have
learned from a popular benchmark.

## What matters in enterprise phone calls

Phone audio is not a clean laboratory input. Codec choices, sampling, carrier routing, speakerphones, headsets, room acoustics and network instability all alter the signal. Callers interrupt, answer with a short “uh-huh,” talk over another person or return from hold music and transfers. One score on curated audio cannot represent all of those conditions.

### Business-critical meaning, not WER alone

Word error rate gives every edit a mathematical role, but it does not
automatically reflect the cost of the error. Transcribing 5 p.m. as 3
p.m., dropping “not,” or changing one digit in an address can break a
workflow more severely than several filler-word errors. Add separate
checks for names, phone numbers, addresses, [appointment](/use-cases/appointment-booking-faqs) times,
negations, intent labels and values passed to tools. Record false
insertions as well as omissions and substitutions.

### Streaming behavior, not batch throughput

A live assistant acts on partial and final transcripts. Partials must be stable enough for downstream logic, a pause must not trigger endpointing too soon, and interruptions should not leave the conversation in the wrong state. Measure time to a stable final segment through the whole conversational system. Fast processing of a long recording is not the same as low turn latency on a call.

### Language variance and code-switching

Multilingual benchmark averages are informative, but they can hide large differences among languages and speaker groups. Break results down by language, dialect or accent, noise class and task. Test the actual switching patterns your callers use—for example, a German sentence containing an English product name—rather than assuming two clean monolingual samples cover the problem.

### Speaker overlap, transfers and diarization

Speaker attribution matters when more than one voice enters the call,
but it is only one layer of quality. Evaluate overlapping speech,
background voices, transferred calls and audio from another device.
Famulor’s separate guide to [speaker
diarization for AI voice agents](https://www.famulor.io/blog/speaker-diarization-for-ai-voice-agents-a-full-guide) covers that specialist subject in
depth; here it belongs as one dimension in a broader production
test.

## What Hacker News signals—and what it does not

The Hacker News discussion is sparse so far. In a September 7
snapshot, the [Muse submission](https://news.ycombinator.com/item?id=49527022)
had four points and one comment. A [MAI-Transcribe-2
thread](https://news.ycombinator.com/item?id=49563026) had two points and three comments including a nested reply.
Practitioners asked whether the model was streaming or batch and what
[pricing](/pricing) would look like after a limited launch offer; one commenter said
they had tried it through Voice Live and linked to Microsoft Learn.

This is a **weak practitioner signal**, not market
validation or consensus. The questions are still useful because they
expose concerns that headline WER leaves unanswered: deployment mode,
the complete interaction path and durable commercial terms. The small
sample cannot establish model quality or adoption. Likewise, the [HN submission about
benchmark optimization](https://news.ycombinator.com/item?id=49589055) had only two points and no comments in the
same snapshot.

## An eight-step production-like STT evaluation

Treat this as an editorial testing protocol, not a universal Famulor benchmark. Each team should set acceptance criteria according to the cost of errors in its own workflow and compare candidates against its own baseline.

1. **Define the call slice.** Specify inbound or outbound,
language mix, normal duration, transfer behavior and business-critical
fields. An appointment line and a technical help desk need different
cases.
2. **Use representative phone audio.** Include accents,
background noise, interruptions, crosstalk, hold audio, short
acknowledgements and the actual telephony route. Studio-microphone
recordings alone are insufficient.
3. **Protect test data.** Prefer synthetic calls or
material for which appropriate use and processing have been established.
Do not upload sensitive production calls to a new service merely to run
a quick comparison.
4. **Measure semantic and operational errors.** Track
critical entities, negations, intents, tool parameters and false
insertions alongside WER. Weight an error according to its consequence
in the workflow.
5. **Observe streaming behavior.** Examine
partial-transcript stability, endpointing, time to final, interruptions
and conversation-level latency. Keep batch throughput distinct from
interactive responsiveness.
6. **Segment the results.** Report language, accent,
noise, scenario and critical-field outcomes separately. Use a blended
score only when its weighting resembles the real call mix.
7. **Run blind review and preserve regressions.** Have
reviewers compare transcripts without model labels. Keep a fixed private
holdout set and add fresh, unseen calls over time.
8. **Pilot with safeguards.** Begin in simulation or
staging, define rollback criteria and verify fallback behavior before
moving production traffic.

You can build a useful scorecard without fabricating a single result. Start with dimensions, then populate observations exclusively from your test run:

| Evaluation dimension | Observation to record | Useful segments | 
|---|---|---|
| Critical entities | Correct or incorrect names, numbers, addresses and times | Language, accent, noise | 
| Negation and intent | Meaning changes, omissions and false insertions | Use case, utterance type | 
| Partial transcripts | Stability and revisions while the caller speaks | Short versus long utterance | 
| Endpointing | Early or late end-of-turn detection | Pause, interruption, overlap | 
| Time to final | Measured system time from audio to stable segment | Phone route, region, load | 
| Diarization | Correct speaker attribution and changes | Two or more speakers, transfer | 
| Tool inputs | Accurate transfer of critical values | Tool and field type | 
| Operations | Availability, failure mode and fallback during the pilot | Provider route, scenario | 

For testing beyond the transcription layer—including dialog logic,
voice output and actions—use the broader guide to [testing
and evaluating an AI voice agent](https://www.famulor.io/blog/how-to-test-and-evaluate-an-ai-voice-agent-in-2026).

## How should teams interpret Muse and MAI today?

For Muse, live voice AI teams should pay particular attention to streaming, endpointing, code-switching and biasing on their own phone samples. Meta’s long-audio and multi-speaker specifications may be relevant to particular workflows, but they do not substitute for tests of crosstalk, transfers and the full audio chain. The Muse web demo states that microphone audio used there is processed for transcription and not stored. That statement applies to that specific demo; any production assessment should review the terms of the chosen access path separately.

For MAI, useful evaluation targets include biasing, automatic language identification, code-switching, timestamps and the documented Voice Live route. Microsoft reports a 5.2% average WER on FLEURS across 60 languages. This is a vendor-reported public-benchmark average, not an expected error rate for German customer calls or any other individual workload. Buyers should also check the service region, data processing terms, contract terms and complete cost path. A time-limited launch offer cannot establish the long-term operating price.

In both cases, verify current provider availability, integration
mode, data requirements and commercial terms before procurement.
Famulor’s evergreen guide to [choosing
a speech-to-text provider](https://www.famulor.io/blog/how-to-choose-the-right-speech-to-text-stt-provider-for-your-ai-voice-agent) explains the broader architectural
criteria.

## What does this mean for a Famulor team?

The current Famulor [Models &
voices documentation](https://docs.famulor.io/assistants/models-and-voices) describes a workspace catalog spanning language
models, speech recognition, text-to-speech and realtime models. Visible
availability depends on the workspace and engine mode. The documentation
recommends retesting important scenarios after a model, voice or
language change, and notes that simulations can reveal pronunciation and
timing differences. Famulor also documents a speech-recognition glossary
for customer, product and proper names; that feature is relevant to
domain-term evaluation, but it does not prove the performance of any new
model.

The product boundary must remain explicit: Famulor’s documentation
index, website index and changelog, checked on September 7, 2026,
**do not document support for Meta Muse Voice Transcribe or
Microsoft MAI-Transcribe-2**. Teams should evaluate only models
that are actually available in their workspace and treat external
launches as market intelligence unless current product documentation
says otherwise. When an available option changes, rerun the same
held-out set instead of transferring conclusions from a different
model.

## Conclusion: the new STT trend is also a testing trend

Muse Voice Transcribe and MAI-Transcribe-2 show that speech recognition is becoming richer than a final text field. Streaming, endpointing, speaker attribution, biasing and language switching are moving closer to the realities of live conversation. At the same time, new benchmark research is a timely reminder that a strong public score does not guarantee generalization to fresh telephone audio.

The defensible decision is therefore two-stage. Use vendor documentation and public rankings to shortlist candidates. Then build a privacy-aware, held-out phone test set, measure business-critical errors and streaming behavior, and pilot with a clear rollback path. That turns a model switch into an evidence-based operational choice rather than a reaction to a headline.

*Sarah Müller writes about voice AI, telephony and the responsible
adoption of new AI models at Famulor. Changeable product and source
information in this article was checked on September 7, 2026.*

Writer at Famulor