{"slug": "how-to-run-nemotron-3-5-transcribe-locally-with-vllm", "title": "How to Run Nemotron 3.5 Transcribe Locally with vLLM", "summary": "Nvidia's open-weight Nemotron 3.5 Transcribe speech model, a 30-billion-parameter mixture-of-experts architecture with roughly 4.3 billion active parameters per token, runs locally through vLLM at just over 75GB of VRAM in BF16 precision and occupies around 67GB of disk, according to a deployment guide. The model handles transcription, speaker diarization, word-level timestamps and translation across roughly 40 languages, with recordings up to an hour long, switching behavior via the text instruction paired with the audio. Nvidia's benchmarks show FP8 and NVFP4 quantization, especially with speculative decoding, push throughput past 2,000 seconds of audio per second of compute, though the guide reports quality varied noticeably on harder languages such as Hebrew and Urdu, and the model ships under an OpenMDW-style license that creators should check before commercial use.", "body_md": "# How to Run Nemotron 3.5 Transcribe Locally with vLLM\n\nDeploy Nvidia's Nemotron 3.5 Transcribe speech model locally with vLLM. Real VRAM, disk, and quantization details for 40-language transcription.\n\n## What is Nemotron 3.5 Transcribe?\n\nNemotron 3.5 Transcribe is an open-weight speech model from Nvidia built for transcription, speaker labeling, and translation across roughly 40 languages, with support for recordings up to an hour long. It uses a mixture-of-experts design with about 30 billion total parameters, only around 4.3 billion of which are active per token, which keeps inference faster and lighter than a dense model of the same size would be. You run it locally through vLLM, feeding it audio plus a text instruction, and it returns verbatim transcripts with speaker tags, word-level timestamps, filler words, and even translated output, all from one model.\n\n## TL;DR\n\n- **Nemotron 3.5 Transcribe** is Nvidia’s open-weight speech model, a 30B-parameter mixture-of-experts architecture with about 4.3B active parameters per inference pass.\n- Running it in full precision (BF16) through **vLLM** requires a GPU with roughly 80GB of VRAM, since observed usage sits just above 75GB.\n- The model needs around **67GB of disk space** , and that footprint can shrink somewhat by reducing the context window.\n- It handles **speaker diarization, verbatim transcription, and timestamping** in a single pass, controlled entirely by changing the text instruction you pair with the audio.\n- Nvidia’s own benchmarks show **quantized formats like FP8 and NVFP4** , especially paired with speculative decoding, push throughput far beyond plain BF16, with the fastest configuration processing over 2,000 seconds of audio per second of compute.\n- Real-world testing across dozens of languages, including lower-quality and accented audio, showed generally strong results, though quality varied noticeably on harder languages like Hebrew and Urdu.\n- The model ships under an **OpenMDW-style license** , which creators should check carefully before using it commercially.\n\n## How do you set up Nemotron 3.5 Transcribe with vLLM?\n\nThe deployment path is straightforward if your hardware clears the bar. vLLM serves the model behind an inference endpoint, and you interact with it by sending audio alongside a plain-text instruction describing what you want back: a clean transcript, a verbatim one with filler words, speaker labels, timestamps, or a translation into another language. The architecture behind that flexibility combines an audio encoder (which processes the raw sound), a speaker encoder (which distinguishes different voices in the same recording), and the Nemotron language model itself, which generates the final text output. A separate timestamp head attaches timing information to each word.\n\nBecause it’s one model handling many output types, you don’t need separate pipelines for diarization, transcription, and translation. Changing the instruction text is enough to switch behavior, which simplifies integration if you’re building this into a larger application rather than running it as a standalone tool.\n\n## What VRAM and disk space does Nemotron 3.5 Transcribe actually need?\n\nRunning the model in full precision via vLLM consumes just over 75GB of VRAM during active inference. That effectively means you need a single GPU with 80GB of VRAM, such as an A100 or H100, to run it comfortably in production without precision tricks. On disk, the full model takes up around 67GB.\n\nIf 80GB of VRAM isn’t available, reducing the context window is one lever that brings memory usage down, though it comes at the cost of how much audio you can process in a single pass. For anyone trying to fit this onto smaller cards, quantization is the more practical route (see below), since it cuts memory and compute requirements more substantially than context trimming alone.\n\n## How fast is it, and do quantized versions help?\n\nSpeed is where Nemotron 3.5 Transcribe stands out. Nvidia’s own benchmarking (run on comparable but not identical hardware to a typical local setup) shows a clear pattern: lower-precision formats consistently outperform full BF16 precision in throughput. Specifically, FP8 and NVFP4 formats, especially when combined with speculative decoding, deliver significantly higher throughput than plain BF16 inference.\n\nIn Nvidia’s best-case configuration, the model processes over 2,000 seconds of audio for every second of compute time. In practical terms, a 30-second audio clip returns a transcript in a fraction of a second. That kind of throughput matters if you’re processing large batches of calls, meetings, or podcast episodes rather than transcribing audio one file at a time, and it’s a big part of why this model is positioned as production-grade rather than just a research release.\n\n## How good is the transcription and translation quality?\n\nTesting across a wide range of languages, including Arabic, German, Spanish, Japanese, Korean, Hindi, and many others, using real-world audio samples (some intentionally low quality or recorded with background noise), the model handled most languages well. It correctly separated overlapping speakers in a phone call transcript, captured filler words and half-finished sentences, and attached accurate timestamps to individual words.\n\n## Other agents start typing. Remy starts asking.\n\nScoping, trade-offs, edge cases — the real work. Before a line of code.\n\nQuality wasn’t perfectly uniform. Harder languages like Hebrew and Urdu showed more noticeable errors, particularly on degraded audio, and some output on those samples was flagged as not fully accurate. That’s a reasonable expectation for a model covering around 40 languages: performance tends to track how much training data exists for each language and dialect, and lower-resource or phonetically complex languages are more likely to show gaps. For the majority of major languages tested, though, transcription and English translation quality looked solid even on noisy or accented source audio.\n\n## Is Nemotron 3.5 Transcribe worth running locally?\n\nFor teams that need transcription with speaker labeling and timestamps baked in, and want to avoid sending audio to a third-party API, this model is a strong option if you have access to an 80GB GPU. The combination of multilingual coverage, verbatim transcription (including fillers and interruptions), and built-in diarization in a single open-weight model is uncommon. Most comparable capability sets require stitching together separate transcription, diarization, and translation tools.\n\nThe catch is hardware cost. An 80GB GPU is not casual home-lab equipment, and renting one on demand is often the more realistic path unless you already have dedicated infrastructure. Quantized formats (FP8, NVFP4) offer a path to running this on less VRAM with faster throughput, though the transcript doesn’t detail exact VRAM figures for those quantized variants, so anyone testing them locally should benchmark their own setup.\n\nThe licensing question also deserves attention before any commercial deployment. The model ships under an OpenMDW-style license, and the specific terms around commercial use should be reviewed directly rather than assumed, since open-weight does not automatically mean unrestricted commercial use.\n\n## Frequently Asked Questions\n\n### How much VRAM does Nemotron 3.5 Transcribe need to run locally?\n\nRunning the model in full BF16 precision through vLLM consumes just over 75GB of VRAM, which in practice means you need a GPU with 80GB of VRAM, such as an A100 or H100, for stable production use.\n\n### How much disk space does the model require?\n\nThe full model needs approximately 67GB of disk space. This can be reduced somewhat by lowering the context window, though that also limits how much audio you can process at once.\n\n### What languages does Nemotron 3.5 Transcribe support?\n\nIt supports around 40 languages, including major European, Asian, and Middle Eastern languages. Testing showed strong results on most languages, with more variable accuracy on harder cases like Hebrew and Urdu, particularly with degraded audio quality.\n\n### Can Nemotron 3.5 Transcribe handle long recordings and multiple speakers?\n\nYes. It supports recordings up to one hour long and includes speaker diarization, meaning it can label different speakers in overlapping or interleaved conversation, along with word-level timestamps.\n\n### Does quantization actually improve performance, and by how much?\n\nNvidia’s benchmarks show FP8 and NVFP4 formats, especially combined with speculative decoding, substantially outperform full BF16 precision in throughput, with the fastest configuration reaching over 2,000 seconds of audio processed per second of compute time.", "url": "https://wpnews.pro/news/how-to-run-nemotron-3-5-transcribe-locally-with-vllm", "canonical_source": "https://www.mindstudio.ai/blog/run-nemotron-transcribe-locally-vllm/", "published_at": "2026-10-30 00:00:00+00:00", "updated_at": "2026-10-30 10:50:40.273305+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-tools", "ai-infrastructure", "natural-language-processing"], "entities": ["Nvidia", "Nemotron 3.5 Transcribe", "vLLM", "A100", "H100", "OpenMDW"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-run-nemotron-3-5-transcribe-locally-with-vllm", "markdown": "https://wpnews.pro/news/how-to-run-nemotron-3-5-transcribe-locally-with-vllm.md", "text": "https://wpnews.pro/news/how-to-run-nemotron-3-5-transcribe-locally-with-vllm.txt", "jsonld": "https://wpnews.pro/news/how-to-run-nemotron-3-5-transcribe-locally-with-vllm.jsonld"}}