NVIDIA dropped Nemotron 3.5 Transcribe on 28 October 2026: an open-weight speech-augmented language model (SALM) that does transcription, speaker attribution, translation, and audio Q&A from a single checkpoint. It activates roughly 4.3B parameters out of a 34B MoE hybrid Mamba-Transformer backbone, handles 40 languages, up to eight speakers in 13 of them, and recordings up to an hour. It sits at the top of the open-weight English and multilingual Open ASR leaderboards. The catch for anyone building a live call or interview assistant is right in the model card: it is offline. No streaming endpoint. You feed it a finished recording.\n\n\n\n## Why the offline design is both the feature and the constraint\n\nConventional ASR systems hard-code one output style at training time. Nemotron 3.5 Transcribe lets you choose with a natural-language instruction: verbatim, written form, speaker-attributed, contextual with entity hints, or translated. The architecture pairs a trainable Transformer speech encoder with a frozen Nemotron 3 Diarization model (Sortformer-style) and feeds both into the Nemotron 3.5 Lightning LLM. Speaker activity is injected every 80 ms so the language model already sees who spoke when. Word-level timestamps come from a lightweight CTC adapter after decoding.\n\nThat consolidation is useful. Instead of running a separate diarizer, aligner, and translator, one forward pass can produce a speaker-labeled transcript and then answer questions about the audio. NVIDIA also released Granary v2, the public data pipeline behind it, and fine-tuning recipes. On a medical speech benchmark they report a large WER drop after domain adaptation in a handful of GPU-hours.\n\nThe offline constraint is deliberate. Long-context MoE inference on hour-long audio benefits from seeing the whole spectrogram and from multi-token prediction / speculative decoding. The published throughput numbers (thousands of seconds of audio per second of wall time on a Blackwell GPU with NVFP4 + DFlash2) are batch numbers, not interactive ones. Time-to-first-token on a live stream is not the optimization target.\n\n## What a live Windows call path actually needs\n\nA desktop app that sits beside Teams, Zoom, or Meet and shows suggestions while the other person is still talking has a different budget. The useful window is the gap between the last speech sample and the moment a reply can be displayed without overlap. That gap is measured in tens to low hundreds of milliseconds, not the seconds required to finalize a multi-minute recording.\n\nThe capture side on Windows is still the same problem it has been for years:\n\n- WASAPI shared mode so the call client keeps its own endpoint.\n- Loopback on the render device for the remote party.\n- A second stream for the local microphone.\n- Resampling to whatever the STT expects (usually 16 kHz mono).\n- Clock alignment between the two tracks so speaker labels stay meaningful.\n\nA minimal loop that would feed an offline model later looks like this (Python sketch; production code uses a thinner WASAPI wrapper or cpal in Rust):\n\n
python\nimport pyaudiowpatch as pyaudio\nimport wave\nimport time\n\np = pyaudio.PyAudio()\n# loopback device for the call\nstream = p.open(format=pyaudio.paInt16,\n channels=1,\n rate=16000,\n input=True,\n frames_per_buffer=320, # 20 ms\n input_device_index=loopback_index)\n\nframes = []\nstart = time.monotonic()\nwhile time.monotonic() - start < 30: # collect a short segment\n frames.append(stream.read(320, exception_on_overflow=False))\n\n# later: write to wav and call the offline model\nwith wave.open(\"segment.wav\", \"wb\") as wf:\n wf.setnchannels(1)\n wf.setsampwidth(2)\n wf.setframerate(16000)\n wf.writeframes(b\"\".join(frames))\n
\n\nTwenty-millisecond chunks keep the capture clock honest. Once the segment is written you can hand it to Nemotron 3.5 Transcribe (via Transformers, NeMo, or a quantized GGUF path) with a prompt such as “Transcribe with speaker labels, then translate the remote speaker into English.” The model will do the heavy lifting. The latency, however, is the sum of segment length + inference + post-processing. For a live overlay that is usually too late.\n\n\n\n## Tradeoffs that show up immediately\n\n| Approach | Latency | Speaker labels | Domain adaptation | Hardware | Cost model |\n|----------|---------|----------------|-------------------|----------|------------|\n| Nemotron 3.5 Transcribe (offline) | seconds (segment + infer) | strong, promptable | open fine-tune | local GPU or cloud | free weights |\n| Streaming commercial STT | 40-200 ms final | varies | prompt or fine-tune | API | per minute / hour |\n| Small local streaming ASR | 50-150 ms | weak or none | limited | CPU/GPU | free |\n\nThe open model wins when you need high-quality offline transcripts, multi-speaker attribution, translation in one pass, or the ability to fine-tune on your own data without sending audio to a third party. It loses when the user is mid-sentence and you need a partial or a final within a few hundred milliseconds to drive an LLM suggestion or an overlay.\n\nA practical hybrid that many desktop tools end up with is:\n\n1. A low-latency streaming STT (or a small local model) for the live path that only produces rough partials and turn-end signals.\n2. An occasional offline pass with something like Nemotron 3.5 Transcribe on the last 30-60 seconds when the call is quiet, to produce a clean speaker-labeled transcript that can be used for later search or for richer context in the next suggestion.\n\nThe streaming path still has to solve the same Windows audio issues: exclusive vs shared mode, default-device changes mid-call, communication-device ducking, and keeping the two tracks time-aligned so the offline model’s diarization matches what the user heard.\n\n## Running the model without a data-center GPU\n\nNVIDIA published BF16, FP8, and NVFP4 checkpoints. The NVFP4 variant is intended for consumer Blackwell cards (RTX 5090 class) and edge devices such as DGX Spark or Jetson. GGUF files exist for NeMo-Speech.cpp style runtimes. Memory footprint drops from roughly 70 GB (BF16) into the low-20 GB range with aggressive quantization, which is the difference between “needs a cloud instance” and “fits on a high-end laptop or desktop next to the call client.”\n\nInference code in Transformers looks roughly like:\n\n
python\nfrom transformers import AutoModelForMultimodalLM, AutoProcessor\n\nmodel_id = \"nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B\"\nprocessor = AutoProcessor.from_pretrained(model_id)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n model_id, device_map=\"auto\", torch_dtype=\"auto\"\n)\n\ninputs = processor.apply_transcription_request(\"segment.wav\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=512)\ntext = processor.decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)\n
\n\nAdd a prompt that requests speaker labels and a translation if that is what the overlay needs. The same checkpoint can later answer “what technical terms were discussed in the last minute?” without a second model.\n\n## Where this leaves a live interview tool\n\nFor a Windows app that helps during job interviews or calls in another language, the offline model is a high-quality second stage, not the primary real-time engine. The primary path still has to be a streaming recognizer that can emit partials fast enough for turn-taking, plus a capture stack that does not inject a bot into the meeting. Dual-track (mic + loopback) capture, short buffers, and careful clock alignment remain the parts that decide whether the suggestion arrives before the other person finishes their sentence.\n\nThe open weights and promptable interface make it straightforward to offer a local high-accuracy mode for users who have the GPU and prefer not to send audio off-device, while still supporting BYOK streaming endpoints for the low-latency path. That split is the practical consequence of an offline speech LLM that is excellent at what it was designed for and simply not designed for the sub-second reaction window of a live call.\n\nI'm building Talkaside, a Windows app for job interviews and calls in another language.\nDrafted with AI assistance from this project's code and logs.\n