How to Fine-Tune Nemotron 3.5 ASR for Your Language, Domain, or Accent NVIDIA released Nemotron 3.5 ASR, a 600M-parameter streaming multilingual speech-to-text model that transcribes 40 language-locales from a single checkpoint with built-in punctuation and capitalization. The model, which succeeds the English-only Nemotron 3 ASR, uses a Cache-Aware FastConformer-RNNT architecture to achieve low latency and high accuracy without redundant audio processing. Available as open weights on Hugging Face, the model can be fine-tuned for specific languages, domains, or accents without API dependencies or per-call billing. Automatic Speech Recognition • Updated • 225 • 47 How to Fine-Tune Nemotron 3.5 ASR for Your Language, Domain, or Accent Enterprise + Article /blog Published June 4, 2026 NVIDIA Nemotron 3.5 ASR https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b , streaming multilingual: a 600M-parameter speech-to-text model that transcribes 40 language-locales from a single checkpoint , in real time , with punctuation and capitalization built in . It is the successor of the popular Nemotron 3 ASR model English only which was released on Hugging Face https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b and as a NIM https://build.nvidia.com/nvidia/nemotron-asr-streaming/modelcard earlier this year. Since its release, Nemotron 3 ASR has been validated by independent benchmarks at Artificial Analysis, where it ranks 2nd in latency among all streaming ASR models https://artificialanalysis.ai/speech-to-text/streaming — with just 0.07 seconds to final transcript after end of speech — and sits in the "most attractive quadrant" of the AA-WER Streaming Index vs. Time to Final Transcription leaderboard, placing it among the best models on the combined accuracy-latency tradeoff. The model uses a Cache-Aware FastConformer-RNNT architecture that streams audio without the redundant recomputation that makes most streaming ASR slow — so you get low latency and high accuracy, not one at the expense of the other. Nemotron 3.5 ASR ships as open weights on Hugging Face — you can inspect, fine-tune, and deploy it without API dependencies or per-call billing. No data leaves your infrastructure unless you choose. And because it's a strong base model, you can fine-tune it for your own language, domain, or accent. The second half of this post walks through exactly how. The problem with multilingual speech recognition today If you've ever built a product that needs to transcribe speech, you've probably hit one of these walls: The polyglot tax. You want to support multiple languages, so you stitch together 40 different models — or 40 different vendor APIs — each with its own quirks, latency profile, and billing. Your infrastructure becomes a museum of one-off integrations. The streaming-vs-accuracy tradeoff. Real-time captioning needs low latency, but most "streaming" ASR systems fake it by re-processing overlapping windows of audio over and over. That burns compute and adds delay. Turn down the latency and accuracy falls off a cliff. The post-processing pipeline. Raw ASR output is often an unpunctuated, lowercase wall of text. You bolt on a second model for punctuation and capitalization, adding yet another moving part. The "known language" assumption. Many systems require you to tell them the language up front. But what about a customer-support line where callers switch between English and Spanish mid-sentence? Nemotron 3.5 ASR was built to collapse all four of those problems into one model. What it does One model, 40 language-locales. A single 600M-parameter checkpoint transcribes English US/GB , Spanish US/ES , German, French FR/CA , Italian, Arabic, Japanese, Korean, Portuguese BR/PT , Russian, Hindi, Turkish, Vietnamese, Dutch, Ukrainian, Polish, Finnish, Mandarin, Czech, Bulgarian, Slovak, Swedish, Croatian, Romanian, Estonian, Danish, Hungarian, Norwegian Bokmål, Norwegian Nynorsk, Hebrew, Greek, Lithuanian, Latvian, Maltese, Slovenian, and Thai. No per-language deployment, no model-swapping. Real-time streaming, done right. The model is built on a Cache-Aware FastConformer encoder. Traditional "buffered" streaming re-processes overlapping chunks of audio at every step, doing the same work many times over. This model instead caches the encoder's internal state and reuses it — every audio frame is processed exactly once, with no overlap. The result is dramatically lower compute and end-to-end latency, with no accuracy penalty. Punctuation and capitalization, natively. The output is production-ready text — proper casing, commas, periods, question marks — straight from the model. No separate punctuation-restoration step. Language conditioning, your choice. You can run it two ways: Tell the model the input language target lang=en-US when you know it — typically the best accuracy. Let the model detect the language target lang=auto when you don't — the model detects the language and transcribes accordingly. How it works the 2-minute version The model has two main pieces: A Cache-Aware FastConformer encoder 24 layers . FastConformer is an efficient evolution of the Conformer architecture with linearly scalable attention. The "cache-aware" part is the streaming magic: the encoder keeps a cache of its self-attention and convolution activations from previous frames, so as new audio arrives it only computes what's genuinely new. Nothing is recomputed. An RNNT Recurrent Neural Network Transducer decoder. RNNT is the workhorse decoder for streaming ASR — it emits text as audio streams in, frame by frame, which is exactly what you want for live transcription. On top of this, the model adds prompt-based language-ID conditioning : a language signal is fed alongside the audio, which lets one set of weights specialize its output to the target language — or, in auto mode, infer the language itself. It was trained on a massive speech data spanning all supported languages, using a blend of public and proprietary data normalized to punctuated, properly-cased text. A knob worth knowing: att context size Streaming ASR is fundamentally a tradeoff between how soon you emit text and how much future audio the model gets to "peek at" before committing. Nemotron ASR exposes this directly through the attention context size : | Attention Context | Chunk Size Latency | Use Case | |---|---|---| 56, 0 | 80ms Ultra-Low | Ultra low latency Voice Agents | 56, 1 | 160ms Low | Interactive Voice Agents, Conversational AI | 56, 3 | 320ms Balanced | Conversational AI, Live caption | 56, 6 | 560ms Medium | High accuracy with reasonable latency | 56, 13 | 1.12s High | Highest accuracy with high latency | The same checkpoint covers the whole spectrum — you choose the operating point at inference time, no retraining required. Try it in minutes The model ships as a NeMo checkpoint. Clone the NeMo branch and point the streaming inference script at your audio: git clone https://github.com/NVIDIA-NeMo/NeMo.git Transcribe with a known language: python ${NEMO ROOT}/examples/asr/asr cache aware streaming/speech to text cache aware streaming infer.py \ model path=${MODEL PATH} \ dataset manifest=${MANIFEST PATH} \ output path=${OUTPUT FOLDER} \ target lang=es-ES \ att context size=" 56,3 " \ strip lang tags=true Or let the model detect the language: python ${NEMO ROOT}/examples/asr/asr cache aware streaming/speech to text cache aware streaming infer.py \ model path=${MODEL PATH} \ dataset manifest=${MANIFEST PATH} \ output path=${OUTPUT FOLDER} \ target lang=auto \ att context size=" 56,3 " \ strip lang tags=true Audio should be mono-channel .wav . The manifest is a standard NeMo JSON-lines file: {"audio filepath": "/path/to/clip.wav", "duration": 4.27, "text": "reference transcript"} Model automatically predicts language tag at the end of each completed sentence, i.e. “This is a test sample.