NVIDIA Releases Nemotron 3 Diarization: A 100M-Parameter Open-Weight Model That Tracks 8 Speakers in Real Time

NVIDIA released Nemotron 3 Diarization, a 100M-parameter open-weight speaker diarization model on Hugging Face under the OpenMDW License 1.1, which permits commercial use and tracks up to 8 speakers, including overlapping voices, in both offline and real-time streaming modes. The model ranked first among 12 systems and 17 configurations in Voice Arena's initial Diarization-Bench results, scoring 14.72% DER across 139 English conversations totaling about 22 hours, versus 19.3% for the next-ranked system, a roughly 24% relative reduction. Against NVIDIA's earlier 4-speaker Streaming Sortformer checkpoint at 1.04 s latency, DER dropped on all 8 evaluation conditions, with relative reductions from 9.0% on CALLHOME-Part2 to 65.2% on NOTSOFAR1 MHM.

NVIDIA has released Nemotron 3 Diarization https://huggingface.co/nvidia/Nemotron-3-Diarization , an open-weight speaker diarization model on Hugging Face. It answers one question about any conversation: who spoke when. The 100M-parameter model tracks up to 8 speakers, including when voices overlap. One checkpoint handles both offline recordings and real-time streaming. Is it deployable? Yes. The weights are released under the OpenMDW License 1.1 https://openmdw.ai/license/1-1/ , which permits commercial use. It runs on Linux through NVIDIA NeMo, using Ampere, Ada Lovelace, Hopper, or Blackwell GPUs. Why Speaker Diarization? Automatic speech recognition ASR gives you the words. It does not tell you who said them. Without attribution, a summarizer cannot tell who made a commitment or who raised an objection. Diarization outputs the time intervals where each speaker is active. Those timestamps combine with ASR output to produce a speaker-attributed transcript. Meeting tools, call analytics, podcast pipelines, and voice-agent memory all depend on this step. What Changed From Streaming Sortformer NVIDIA’s earlier Streaming Sortformer https://arxiv.org/abs/2507.18446 checkpoint, diar streaming sortformer 4spk-v2.1 , supported 4 speakers. Nemotron 3 Diarization doubles that limit to 8. According to NVIDIA’s announcement https://x.com/NVIDIAAI/status/2102775666366435450 , the target is messy multi-party audio where people talk at once. How the Architecture Works The model accepts 16 kHz, single-channel audio in .wav, .flac, .opus, or .mp3 format. It converts the audio into Mel-spectrogram features with a 10 ms step. The features are stacked by a factor of 8, which produces 80 ms encoder frames. A 31-layer Transformer encoder with rotary positional embeddings RoPE processes those frames. A Conv1D layer then upsamples the predictions back to 10 ms resolution. The output is a T, 8 tensor of per-speaker activity probabilities. This design handles overlap directly. If 2 people talk at the same time, 2 channels activate in the same frame. The model follows the Sortformer https://arxiv.org/abs/2409.06656 approach of ordering speakers by arrival time. The first new voice takes channel 1, the next takes channel 2, and so on. This keeps labels stable across streaming chunks, so the model does not have to re-match speakers to channels for every chunk. Streaming uses 2 memory mechanisms. The Arrival-Order Speaker Cache AOSC keeps speaker information from earlier chunks. A FIFO queue supplies recent frame context. The labels are anonymous, and mapping them to real identities is left to downstream applications. 4 Latency Operating Points Input-buffer latency equals chunk + right context × 80 ms. The table uses DIHARD III full-set DER and batch-32 compiled throughput from the model card https://huggingface.co/nvidia/Nemotron-3-Diarization . | Configuration | Buffer latency | DIHARD III DER | RTFx batch 32, compiled | |---|---|---|---| | Offline style | 30.4 s | 12.73% | 15,113× | | Low latency | 1.04 s | 13.18% | 865× | | Very low latency | 0.64 s | 13.28% | 579× | | Ultra-low latency | 0.32 s | 13.55% | 292× | This latency excludes compute, networking, and ASR time. The model can technically run with an 80 ms buffer, but 0.32 s is the lowest recommended setting. Benchmark Results In Voice Arena’s initial Diarization-Bench results, the model ranked first among 12 systems and 17 configurations. The test covered 139 English conversations totaling about 22 hours. It scored 14.72% DER against 19.3% for the next-ranked system, roughly a 24% relative reduction. NVIDIA notes these results may change once Voice Arena completes its Version 1 evaluation. Against the 4-speaker baseline at 1.04 s latency, DER dropped on all 8 evaluation conditions. Relative reductions ranged from 9.0% on CALLHOME-Part2 to 65.2% on NOTSOFAR1 MHM. The unweighted mean across the 8 conditions was 41.0%. There is one regression. On 2-speaker CALLHOME at 30.4 s, DER rose from 5.68% to 5.98%. Full-set CALLHOME-Part2 still improved from 10.32% to 9.10%. Throughput also jumped. At 30.4 s, the model reached 15,113× RTFx versus 2,619× for the baseline. The tests used BF16 on an NVIDIA RTX PRO 5000 with torch.compile . These are batched numbers, not single-stream application latency. Training Data Training combined about 10,000 hours of real conversations with 82,611 hours of simulated multi-talker mixtures. The mix included real-world multi-speaker audio licensed from David AI https://www.withdavid.ai/ . Adding the David AI data cut compound DER from 11.19% to 10.42%. The licensed source audio for the simulated mixtures spans 21 languages. Getting Started Install NVIDIA NeMo Speech https://github.com/NVIDIA-NeMo/Speech with Python 3.12 or later: uv pip install 'nemo-toolkit asr ' python from nemo.collections.asr.models import SortformerEncLabelModel diar model = SortformerEncLabelModel.from pretrained "nvidia/Nemotron-3-Diarization" diar model.eval segments = diar model.diarize audio= "conversation.wav" , batch size=1 Output segments take the form start end speaker id . To get the words as well, pair the model with Parakeet TDT 0.6B v3 https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3 using the ASR integration guide https://huggingface.co/nvidia/Nemotron-3-Diarization/blob/main/ASR INTEGRATION GUIDE.md . The live demo Space https://huggingface.co/spaces/nvidia/nemotron-diarization offers synthetic conversations, a live mic, a multilingual live mic, and audio upload. For production, NVIDIA lists Baseten https://www.baseten.co/library/ and DigitalOcean https://cloud.digitalocean.com/model-studio/explore-models/model-catalog?usecase=audio%20 . On-device support is available through Argmax Pro SDK 3 https://www.argmaxinc.com/blog/argmax-sdk-3 . The model is not yet available through Hugging Face Inference Providers. The model has limits. Recordings with more than 8 speakers can produce missed or misassigned speech. Heavy noise, reverberation, and far-field capture can also raise error rates. Interactive Explainer Key Takeaways - 100M-parameter open-weight model tracks up to 8 overlapping speakers. - One checkpoint covers everything from offline 30.4 s to ultra-low 0.32 s streaming. - Ranked 1 on Voice Arena’s initial Diarization-Bench at 14.72% DER. - Averages a 41.0% relative DER reduction versus Streaming Sortformer at 1.04 s. - The OpenMDW-1.1 license permits commercial use; the model runs on NVIDIA GPUs through NeMo. Check out the Model Weights https://huggingface.co/nvidia/Nemotron-3-Diarization , Technical Blog https://huggingface.co/blog/nvidia/nemotron-diarization , and Demo https://huggingface.co/spaces/nvidia/nemotron-diarization . All credit goes to the researcher of this project. Also, feel free to follow us on Twitter https://x.com/intent/follow?screen name=marktechpost and don’t forget to join our 150k+ML SubReddit https://www.reddit.com/r/machinelearningnews/ and Subscribe to our Newsletter https://magic.beehiiv.com/v1/f5e63dd4-5653-4f09-83e2-321a8b1ba526?email={{email}} . Wait are you on telegram? now you can join us on telegram as well. https://t.me/machinelearningresearchnews Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us https://forms.gle/MJjjVDPS7whH8Ngs6 Asif Razzaq is the CEO of Marktechpost AI Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.