NVIDIA just open-sourced Nemotron-Labs-TwoTower — a diffusion language model that generates text 2.42x faster than the autoregressive model it was built from, without retraining a single original weight. The weights landed on Hugging Face on July 1–2, 2026. If your team is spending real GPU budget on 30B-scale inference, this deserves immediate attention.
One Model, Two Towers, No Retraining #
The key idea behind Nemotron TwoTower is deceptively clean. NVIDIA starts with an existing open-weight backbone — Nemotron-3-Nano-30B-A3B, a 30B hybrid model with 23 Mamba-2 layers, 6 self-attention layers, and 23 MoE layers. It then creates two copies and assigns them completely different jobs.
The first copy, the AR context tower, stays completely frozen. It processes your prompt and all previously committed tokens the same way it always has, building KV caches and Mamba states in the background. The second copy, the diffusion denoiser tower, gets trained on roughly 2.1 trillion tokens — a small fraction of the 25 trillion used in the original pretraining. Its job: take a block of 16 masked tokens and iteratively resolve them in parallel, cross-attending to the frozen tower’s cached representations at every step.
Instead of committing one token per forward pass like standard autoregressive decoding, TwoTower commits multiple tokens per denoising step — especially early in generation when most positions are easy to resolve with high confidence. Harder positions get more refinement passes. Easy ones get committed immediately. That adaptive commitment pattern is where the 2.42x speedup comes from. According to the TwoTower paper on arxiv, the first denoising step alone commits the largest number of tokens on average.
Nemotron TwoTower Benchmark Numbers: Mostly Good, One Honest Caveat #
At default settings — confidence threshold γ=0.8, block size S=16, measured on 2× H100 — Nemotron-TwoTower retains 98.7% of the autoregressive baseline quality at 2.42x wall-clock throughput. For general knowledge and commonsense tasks, the quality drop is essentially zero. Multilingual tasks hold up similarly well. Some commonsense benchmarks actually improve slightly over the AR baseline.
However, code is where you need to be honest with yourself. HumanEval drops from 79.27 to 75.58 — a 3.7-point slide that is real and noticeable. Math tasks show similar, task-dependent degradation. The reason is structural: code and math require sequential precision where early tokens tightly constrain later ones, and parallel block generation does not fully capture that dependency. NVIDIA ships a full AR mode for exactly this reason, and it is the right call to use it for critical code paths.
The Block Size Rule You Must Not Break #
This is the detail that most TwoTower coverage glosses over, and it will break your results if you miss it. The denoiser was trained at block size S=16. You can safely run inference at S=8 — there is a slight quality gain and modest throughput reduction — but exceeding the training block size is catastrophic. Push inference to S=64 and HumanEval collapses from 76.40 to 19.85. GSM8K drops to 2.20. The model is not broken; it was simply never trained to denoise 64-token spans from a cold start, and the evaluation numbers reflect that precisely.
Default recommendation: use S=16 or lower. Never increase block size above training block size.
Three Modes, Three Use Cases #
TwoTower ships with three inference modes. Choosing deliberately based on your task matters more than the architecture does.
Diffusion mode (default, γ=0.8, S=16)— 2.42x throughput, 98.7% quality. Use this for chat, retrieval-augmented generation, summarization, and anything where a sub-2% quality trade-off is acceptable in exchange for the speed gain.Lower confidence (γ=0.6)— pushes past 3x throughput at roughly 95% quality retained. Useful for bulk draft generation or pipelines with built-in review steps downstream.Full AR mode— standard one-token-at-a-time decoding at full baseline quality. Use for critical code generation, precise mathematical reasoning, or any production path where correctness cannot be traded against a confidence threshold.
Get It Now — With Eyes Open on the Ecosystem #
The Nemotron-TwoTower collection on Hugging Face includes weights, training code built on Megatron-LM, and the full paper. Hardware requirements are specific: 2× H100 80GB or 2× A100 80GB, running BF16, at roughly 59GB per GPU for combined weights. The community has already produced a 4-bit TQ4_1S quantized version that fits on a single GPU at around 38GB — useful for experimentation before committing to the dual-GPU production setup.
The ecosystem gap worth knowing: there is currently no vLLM support and no llama.cpp support for TwoTower’s diffusion path. The model landed days ago and the inference kernel work will come, but it is not here yet. For now, inference runs through Megatron-LM scripts in the official repo. This is the right moment to test and benchmark rather than drop it into production — though teams running open-source inference infrastructure already have the tooling to experiment.
What TwoTower actually demonstrates — beyond its own benchmark numbers — is that diffusion LLMs are now practical at 30B scale with hybrid MoE architectures. Previous work topped out at 1.7B. NVIDIA’s result, and the fact that you adapt an existing AR model without full retraining by training only the denoiser on 2.1T tokens, fundamentally changes the economics for teams weighing inference throughput against model quality. The trade-off is quantified, the weights are public, and the block-size constraint is documented. Start there.