{"slug": "nvidia-releases-nemotron-3-diarization-a-100m-parameter-open-weight-model-that-8", "title": "NVIDIA Releases Nemotron 3 Diarization: A 100M-Parameter Open-Weight Model That Tracks 8 Speakers in Real Time", "summary": "NVIDIA released Nemotron 3 Diarization, a 100M-parameter open-weight speaker diarization model on Hugging Face under the OpenMDW License 1.1, which permits commercial use and tracks up to 8 speakers, including overlapping voices, in both offline and real-time streaming modes. The model ranked first among 12 systems and 17 configurations in Voice Arena's initial Diarization-Bench results, scoring 14.72% DER across 139 English conversations totaling about 22 hours, versus 19.3% for the next-ranked system, a roughly 24% relative reduction. Against NVIDIA's earlier 4-speaker Streaming Sortformer checkpoint at 1.04 s latency, DER dropped on all 8 evaluation conditions, with relative reductions from 9.0% on CALLHOME-Part2 to 65.2% on NOTSOFAR1 MHM.", "body_md": "NVIDIA has released [Nemotron 3 Diarization](https://huggingface.co/nvidia/Nemotron-3-Diarization), an open-weight speaker diarization model on Hugging Face. It answers one question about any conversation: who spoke when. The 100M-parameter model tracks up to 8 speakers, including when voices overlap. One checkpoint handles both offline recordings and real-time streaming.\n\n**Is it deployable?** Yes. The weights are released under the [OpenMDW License 1.1](https://openmdw.ai/license/1-1/), which permits commercial use. It runs on Linux through NVIDIA NeMo, using Ampere, Ada Lovelace, Hopper, or Blackwell GPUs.\n\n## **Why Speaker Diarization?**\n\nAutomatic speech recognition (ASR) gives you the words. It does not tell you who said them. Without attribution, a summarizer cannot tell who made a commitment or who raised an objection.\n\nDiarization outputs the time intervals where each speaker is active. Those timestamps combine with ASR output to produce a speaker-attributed transcript. Meeting tools, call analytics, podcast pipelines, and voice-agent memory all depend on this step.\n\n## **What Changed From Streaming Sortformer**\n\nNVIDIA’s earlier [Streaming Sortformer](https://arxiv.org/abs/2507.18446) checkpoint, `diar_streaming_sortformer_4spk-v2.1`, supported 4 speakers. Nemotron 3 Diarization doubles that limit to 8. According to [NVIDIA’s announcement](https://x.com/NVIDIAAI/status/2102775666366435450), the target is messy multi-party audio where people talk at once.\n\n## **How the Architecture Works**\n\nThe model accepts 16 kHz, single-channel audio in .wav, .flac, .opus, or .mp3 format. It converts the audio into Mel-spectrogram features with a 10 ms step. The features are stacked by a factor of 8, which produces 80 ms encoder frames.\n\nA 31-layer Transformer encoder with rotary positional embeddings (RoPE) processes those frames. A Conv1D layer then upsamples the predictions back to 10 ms resolution. The output is a `[T, 8]` tensor of per-speaker activity probabilities.\n\nThis design handles overlap directly. If 2 people talk at the same time, 2 channels activate in the same frame.\n\nThe model follows the [Sortformer](https://arxiv.org/abs/2409.06656) approach of ordering speakers by arrival time. The first new voice takes channel 1, the next takes channel 2, and so on. This keeps labels stable across streaming chunks, so the model does not have to re-match speakers to channels for every chunk.\n\nStreaming uses 2 memory mechanisms. The Arrival-Order Speaker Cache (AOSC) keeps speaker information from earlier chunks. A FIFO queue supplies recent frame context. The labels are anonymous, and mapping them to real identities is left to downstream applications.\n\n## **4 Latency Operating Points**\n\nInput-buffer latency equals (chunk + right context) × 80 ms. The table uses DIHARD III full-set DER and batch-32 compiled throughput from the [model card](https://huggingface.co/nvidia/Nemotron-3-Diarization).\n\n| Configuration | Buffer latency | DIHARD III DER | RTFx (batch 32, compiled) | \n|---|---|---|---|\n| Offline style | 30.4 s | 12.73% | 15,113× | \n| Low latency | 1.04 s | 13.18% | 865× | \n| Very low latency | 0.64 s | 13.28% | 579× | \n| Ultra-low latency | 0.32 s | 13.55% | 292× | \n\nThis latency excludes compute, networking, and ASR time. The model can technically run with an 80 ms buffer, but 0.32 s is the lowest recommended setting.\n\n## **Benchmark Results**\n\nIn Voice Arena’s initial Diarization-Bench results, the model ranked first among 12 systems and 17 configurations. The test covered 139 English conversations totaling about 22 hours. It scored 14.72% DER against 19.3% for the next-ranked system, roughly a 24% relative reduction. NVIDIA notes these results may change once Voice Arena completes its Version 1 evaluation.\n\nAgainst the 4-speaker baseline at 1.04 s latency, DER dropped on all 8 evaluation conditions. Relative reductions ranged from 9.0% on CALLHOME-Part2 to 65.2% on NOTSOFAR1 MHM. The unweighted mean across the 8 conditions was 41.0%.\n\nThere is one regression. On 2-speaker CALLHOME at 30.4 s, DER rose from 5.68% to 5.98%. Full-set CALLHOME-Part2 still improved from 10.32% to 9.10%.\n\nThroughput also jumped. At 30.4 s, the model reached 15,113× RTFx versus 2,619× for the baseline. The tests used BF16 on an NVIDIA RTX PRO 5000 with `torch.compile()`. These are batched numbers, not single-stream application latency.\n\n## **Training Data**\n\nTraining combined about 10,000 hours of real conversations with 82,611 hours of simulated multi-talker mixtures. The mix included real-world multi-speaker audio licensed from [David AI](https://www.withdavid.ai/). Adding the David AI data cut compound DER from 11.19% to 10.42%. The licensed source audio for the simulated mixtures spans 21 languages.\n\n## **Getting Started**\n\n**Install [NVIDIA NeMo Speech](https://github.com/NVIDIA-NeMo/Speech) with Python 3.12 or later:**\n\n```\nuv pip install 'nemo-toolkit[asr]'\npython\nfrom nemo.collections.asr.models import SortformerEncLabelModel\ndiar_model = SortformerEncLabelModel.from_pretrained(\"nvidia/Nemotron-3-Diarization\")\ndiar_model.eval()\nsegments = diar_model.diarize(audio=[\"conversation.wav\"], batch_size=1)\n```\n\nOutput segments take the form `start end speaker_id`. To get the words as well, pair the model with [Parakeet TDT 0.6B v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) using the [ASR integration guide](https://huggingface.co/nvidia/Nemotron-3-Diarization/blob/main/ASR_INTEGRATION_GUIDE.md).\n\nThe [live demo Space](https://huggingface.co/spaces/nvidia/nemotron-diarization) offers synthetic conversations, a live mic, a multilingual live mic, and audio upload. For production, NVIDIA lists [Baseten](https://www.baseten.co/library/) and [DigitalOcean](https://cloud.digitalocean.com/model-studio/explore-models/model-catalog?usecase=audio%20). On-device support is available through [Argmax Pro SDK 3](https://www.argmaxinc.com/blog/argmax-sdk-3). The model is not yet available through Hugging Face Inference Providers.\n\nThe model has limits. Recordings with more than 8 speakers can produce missed or misassigned speech. Heavy noise, reverberation, and far-field capture can also raise error rates.\n\n## **Interactive Explainer**\n\n## **Key Takeaways**\n\n- 100M-parameter open-weight model tracks up to 8 overlapping speakers.\n- One checkpoint covers everything from offline (30.4 s) to ultra-low 0.32 s streaming.\n- Ranked #1 on Voice Arena’s initial Diarization-Bench at 14.72% DER.\n- Averages a 41.0% relative DER reduction versus Streaming Sortformer at 1.04 s.\n- The OpenMDW-1.1 license permits commercial use; the model runs on NVIDIA GPUs through NeMo.\n\nCheck out the [**Model Weights**](https://huggingface.co/nvidia/Nemotron-3-Diarization), [** Technical Blog**](https://huggingface.co/blog/nvidia/nemotron-diarization), and [** Demo**](https://huggingface.co/spaces/nvidia/nemotron-diarization). All credit goes to the researcher of this project. Also, feel free to follow us on **[Twitter](https://x.com/intent/follow?screen_name=marktechpost)** and don’t forget to join our **[150k+ML SubReddit](https://www.reddit.com/r/machinelearningnews/)** and Subscribe to **[our Newsletter](https://magic.beehiiv.com/v1/f5e63dd4-5653-4f09-83e2-321a8b1ba526?email={{email}})**. Wait! are you on telegram? [now you can join us on telegram as well.](https://t.me/machinelearningresearchnews)\n\nNeed to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? [Connect with us](https://forms.gle/MJjjVDPS7whH8Ngs6)\n\nAsif Razzaq is the CEO of Marktechpost AI Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.", "url": "https://wpnews.pro/news/nvidia-releases-nemotron-3-diarization-a-100m-parameter-open-weight-model-that-8", "canonical_source": "https://www.marktechpost.com/2026/09/23/nvidia-releases-nemotron-3-diarization/", "published_at": "2026-09-23 18:17:14+00:00", "updated_at": "2026-09-23 18:31:59.482461+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "natural-language-processing", "ai-products"], "entities": ["NVIDIA", "Nemotron 3 Diarization", "Hugging Face", "Streaming Sortformer", "Voice Arena", "OpenMDW License 1.1", "NVIDIA NeMo", "DIHARD III"], "alternates": {"html": "https://wpnews.pro/news/nvidia-releases-nemotron-3-diarization-a-100m-parameter-open-weight-model-that-8", "markdown": "https://wpnews.pro/news/nvidia-releases-nemotron-3-diarization-a-100m-parameter-open-weight-model-that-8.md", "text": "https://wpnews.pro/news/nvidia-releases-nemotron-3-diarization-a-100m-parameter-open-weight-model-that-8.txt", "jsonld": "https://wpnews.pro/news/nvidia-releases-nemotron-3-diarization-a-100m-parameter-open-weight-model-that-8.jsonld"}}