Nemotron 3.5 Transcribe on one 32 GB Blackwell card: the prompt is your API, so test it

A developer benchmarked NVIDIA's Nemotron 3.5 Transcribe, released 28 October under the OpenMDW-1.1 license, by running its 21.6 GB NVFP4 model on a single 32 GB RTX PRO 4500 Blackwell card via vLLM and NVIDIA's NeMo Speech plugin, exposing it on the OpenAI-compatible /v1/audio/transcriptions route. On 40 Open ASR Leaderboard clips the model scored 6.82% word error rate, between Bark 3.0 Pro (6.28%) and Qwen3-ASR-1.7B (7.03%), but degraded to 14.62% on crosstalk-heavy AMI meeting audio. Because a single prompt sentence selects verbatim, clean or speaker-tagged output, the tester found the clean style still emitted 50 <spk: tags in three minutes and recommended per-style regression checks such as filler counts and tag absence.

Most speech-to-text stacks bolt features on after the model: diarization working out who said what , then a cleanup pass for the "um"s. NVIDIA's Nemotron 3.5 Transcribe moves those choices into the request. You send the audio plus one sentence, and the sentence decides whether you get every filler, a clean read, or speaker turns with a time on every word. The bugs move with it. NVIDIA released it on 28 October under the OpenMDW-1.1 license https://openmdw.ai/license/1-1/ , with an open serving recipe: vLLM, the open-source engine that loads a model on a GPU and answers requests, plus NVIDIA's NeMo Speech https://github.com/NVIDIA-NeMo/Speech plugin. We took the smallest of its three files, 21.6 GB in a format called NVFP4 that needs a Blackwell card, and ran it on one 32 GB RTX PRO 4500 Blackwell on our cloud. Every figure is from the full Nemotron 3.5 Transcribe test on one 32 GB card https://inferencekey.com/research/nemotron-transcribe-instructable-asr-one-gpu . We wrapped it in a custom backend, a short Python file telling our platform how to install and run a model on a GPU machine. It answers on the standard OpenAI speech route, /v1/audio/transcriptions , like our Qwen3-ASR deployment, so existing client code keeps its shape. Only the prompt is new: out = client.audio.transcriptions.create model="default", file=open "meeting.ogg", "rb" , prompt="speakers", timestamp granularities= "word" speakers is a style name the backend swaps for a full instruction; free text also works. Word timings arrive in a separate ctc timestamps object: start, end and speaker for each word. On the opening three minutes of a PriMock57 https://github.com/babylonhealth/primock57 doctor-patient consultation, verbatim and clean each took 4.1 s of GPU time, and speakers with word timings 4.4 s for 561 timed words. The clean style asks the model to "omit fillers and accidental repetitions; keep the remaining spoken wording unchanged". It closes with "Output: transcript only" and never mentions tags. The fillers did disappear. Yet the output carried 50 <spk: tags across three minutes, and the false start "Hello, how." survived. Code expecting plain prose must strip them, or use the speakers style on purpose. When a sentence picks the format, each style is a code path that needs tests. Cheap ones: no <spk: in clean output, a filler count in verbatim. Verbatim held on to 33 of the 34 "um" and "uh" in the human reference. A compliance team should measure that on its own audio. The write-up prints the exact verbatim, clean and speaker instructions we sent https://inferencekey.com/research/nemotron-transcribe-instructable-asr-one-gpu , so you can rerun those checks. Accuracy here is word error rate, the share of words wrong against a human transcript. On 40 short English clips from the Open ASR Leaderboard data https://huggingface.co/datasets/hf-audio/open-asr-leaderboard , Nemotron scored 6.82%. Scored the same way on those clips, Bark 3.0 Pro and Qwen3-ASR-1.7B in our earlier speech-to-text test https://inferencekey.com/research/underdog-bark-speech-to-text-test landed at 6.28% and 7.03%. In between: you gain output flexibility, not a lower error rate. The average hides a spread. On audiobook speech it was the best of the three at 0.94%. On AMI, recorded office meetings full of crosstalk, it dropped to 14.62%: worse than Bark 11.54% , better than Qwen3-ASR 16.92% . Meeting products: test your own recordings first. The sample is modest: 924 words of clips, a three-minute and a five-minute recording, one request at a time. NVIDIA's 4.71% mean error across nine English sets is its run of the largest version, which we did not repeat. The 2,238 seconds of audio per second on its model page https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B comes from a packaged engine that is not open source yet, and NVIDIA expects the open path to be roughly two times slower. We did not measure throughput or parallel requests. vLLM's --gpu-memory-utilization flag sets what share of the card's memory it may take. Lowering it on the 32 GB card looks like a quick 24 GB test. It is not. With a 23.5 GiB budget, the card still used 25,712 MiB. Only a real 24 GB slice is evidence. That slice, from an RTX PRO 6000 Blackwell, gets a quarter of its compute. Two speed tricks had to go: speculative decoding drafting several words ahead and checking them in one pass and CUDA graphs GPU work recorded once and replayed . With a 16,384-token context and 2 requests at once, it matched the 6.82%, but the three-minute speaker run took 35.3 s instead of 4.4. With speculative decoding on, memory ran out. NVFP4 needs Blackwell, so an RTX 4090 or anything older is out. MAX JOBS=6 . vllm serve kept 25 to 29 GB and the next start failed. Worked around with PR SET PDEATHSIG and a stale pid file; the platform fix SIGTERM with a grace period is still to file. NVIDIA built the model, the NeMo Speech plugin and the serving recipe. The vLLM project built the engine and the FlashInfer team the prebuilt kernels. SGLang-Omni provides an alternative serving path. Babylon Health published PriMock57, hf-audio the test clips, and we spotted the model in an AI Search news video. Our part was the backend, the deployment and the measurements. The flags, full tables, VRAM ladder and an SDK example that runs all three styles on your recording are in the Nemotron 3.5 Transcribe research write-up on inferencekey.com https://inferencekey.com/research/nemotron-transcribe-instructable-asr-one-gpu .