NVIDIA Nemotron-Labs TwoTower: 2.42x Faster Inference, No Retraining Required

NVIDIA released the 30B-parameter Nemotron-Labs TwoTower language model on July 1 that achieves 2.42x faster inference than its autoregressive baseline without retraining from scratch, by splitting a pretrained LLM into a frozen context encoder and a trainable diffusion denoiser. The weights are available on Hugging Face under a commercial license, supporting three generation modes: autoregressive, diffusion (requiring two GPUs), and self-speculation that yields up to 6x tokens per forward pass.

NVIDIA released a 30B-parameter language model on July 1 that runs 2.42x faster than its autoregressive baseline — without retraining from scratch. Nemotron-Labs TwoTower https://huggingface.co/nvidia/Nemotron-Labs-TwoTower-30B-A3B-Base-BF16 achieves this by splitting a pretrained LLM into two cooperating towers: one frozen AR context encoder, one trainable diffusion denoiser. The weights are live on Hugging Face under a commercial license. There are caveats — but so is the underlying technique worth understanding. What Diffusion Decoding Actually Means Standard autoregressive LLMs generate one token at a time, left-to-right, committing to each decision before seeing what comes next. That sequential dependency is also the ceiling on throughput — you cannot parallelize what is structurally serial. Diffusion LLMs take a different path. Instead of predicting the next token, they start with a block of masked tokens and iteratively refine the entire block in parallel. Because the model processes all positions simultaneously, it has bidirectional context within each block — it can see what it is about to write before fully committing. The practical result: more tokens per forward pass, and better coherence for tasks requiring global constraint satisfaction, like code refactoring or structured output generation. The Retrofit That Changes the Equation The interesting thing about TwoTower is not just that it uses diffusion decoding — it is that NVIDIA built it on top of an existing pretrained model rather than training one from scratch. The TwoTower paper https://arxiv.org/abs/2606.26493 describes splitting the Nemotron-3-Nano-30B-A3B weights into two copies: Context tower frozen : Runs the original AR model causally over the prompt and committed tokens. Produces KV caches and Mamba-2 states as context. Denoiser tower trainable : Attends to those cached states while denoising a block of masked output tokens with bidirectional attention within the block. The denoiser was trained on roughly 2.1 trillion tokens — compared to the 25 trillion token full pretraining run. The quality retention number 98.7% of the AR baseline suggests most of the model’s knowledge stays intact because the frozen tower carries it. You are teaching the denoiser to decode differently, not learning from the ground up. This retrofit paradigm is the real story. If it holds up, it means any well-pretrained AR model — Llama, Mistral, Qwen — could theoretically get a diffusion decoder grafted on without starting over. Three Modes From One Checkpoint TwoTower ships a single set of weights that supports three generation modes, selectable by configuration: Autoregressive mode: Standard generation, single 80GB GPU. Use this if you want Nemotron Nano quality with no new infrastructure. Diffusion mode: Block-wise parallel generation across 2 GPUs ~59GB BF16 each . This is where the 2.42x throughput gain applies, at 98.7% quality retention. Self-speculation mode: Diffusion drafts token candidates; the AR tower verifies them. Up to 6x tokens per forward pass in tested configurations — no separate draft model required. NVIDIA provides a place towers on devices utility to distribute the two towers across GPU devices. The NVIDIA diffusion LLM blog post https://huggingface.co/blog/nvidia/nemotron-labs-diffusion covers the full deployment setup. The Hardware Reality Diffusion mode needs two GPUs. The BF16 checkpoint weighs roughly 59GB per tower, making two 80GB H100s the comfortable baseline. If you only have one 80GB GPU, AR mode still works. If you want to reduce memory further, the community has already published an NVFP4 quantized version https://huggingface.co/syscall42/nemotron-twotower-nvfp4 — NVIDIA’s 4-bit float format delivers about 1.8x lower memory footprint than FP8, which opens the door to 2x A100 40GB setups. For context: Mercury 2 from Inception Labs https://www.inceptionlabs.ai/models hits 1,009 tokens per second as a commercial API on Blackwell hardware. TwoTower is not competing on raw speed — it is competing on open weights, quality preservation, and the ability to self-host without a commercial dependency. The Limitation You Must Know Before Deploying The released checkpoint is a base model. It has not been instruction-tuned or aligned for chat, coding, or any conversational use case. NVIDIA’s documentation states that “building production models would require fine-tuning and safety alignment as subsequent steps.” Do not ship this directly to end users. This is a research artifact and a foundation for fine-tuning, not a drop-in replacement for an existing assistant model. If your team can do supervised fine-tuning on a 30B model, TwoTower is a compelling starting point. If not, wait for NVIDIA to release an instruction-tuned variant — which looks likely in Q3 or Q4 2026 given the trajectory of the Nemotron-Labs family. Who Should Benchmark This Now If you operate inference infrastructure at any meaningful scale, the math is worth running. A 2.42x throughput gain on existing hardware translates to roughly 58% fewer GPU-hours for the same output volume. The self-speculation path pushes that further, and the NVFP4 quantization makes the hardware bar more accessible without sacrificing the quality numbers. The diffusion LLM ecosystem is where open-source AR was in early 2024: real, fast-improving, and underserved by production tooling. TwoTower’s retrofit approach may be the fastest path from pretrained AR quality to diffusion speed — and that is worth paying attention to now, before the instruction-tuned version ships.