Teams customize their models to hit their targets for latency, speed, memory, and compute. With the open NVIDIA Nemotron family of models, developers can find the right-sized model for their needs.
The new Nemotron 3.5 Lightning NVFP4 checkpoint, for example, preserves accuracy while unlocking up to 4x faster throughput. It’s compressed down to 22 GB from the 66 GB full precision checkpoint by quantizing many of its weights to 4 bits.
To compress models to NVFP4, post-training quantization (PTQ) is a common method that covers most needs. But if you want to reach high throughput with tighter memory, more aggressive quantization is needed. Quantization-aware distillation (QAD) is an optimal choice in this case. Training Nemotron 3.5 Lightning to adapt to quantization noise with QAD produced an NVFP4 checkpoint that uses less memory and delivers higher throughput while preserving accuracy.
This post demonstrates how QAD improves the Nemotron 3.5 Lightning model using NVIDIA Model Optimizer. We walk through the entire training pipeline, from the initial PTQ phase to the final distillation and evaluation. We show that QAD recovers accuracy degradation from aggressive quantization. Even with more conservative configurations, QAD consistently outperforms PTQ on agentic benchmarks, ensuring high quality while reducing memory usage**. **
What is quantization-aware distillation? #
QAD uses the original full-precision model (teacher) to teach the quantized model (student). First, create a quantized model by running PTQ on the full-precision model. Then distill the frozen BF16 model into the quantized model using a KL divergence loss comparing the teacher’s and student’s logits.
Figure 1 shows the two-stage QAD process used to build the Nemotron 3.5 Lightning NVFP4 checkpoint. The full-precision BF16 model serves as the frozen teacher and is also the starting point for Stage 1, a PTQ pass that quantizes weights to W4A16 to produce the quantized student. In Stage 2, the student is trained with QAD, running its forward pass through simulated quantization while a distillation loss aligns it with the teacher, yielding the final NVFP4 checkpoint with accuracy recovered close to baseline.
The quantization-aware distillation process #
Follow the steps below to execute the QAD process.
Step 1: Post-training quantization
The first stage of QAD is running PTQ to produce a quantized checkpoint (student). Since we plan to run QAD, more aggressive quantization can be performed. For Nemotron 3.5 Lightning, we found that quantizing the Mamba linear layers to a more aggressive W4A16, rather than FP8, unlocked higher throughput without a large drop in accuracy.
Normally, a median accuracy recovery of over 99% is targeted when performing PTQ only. When combined with QAD a target 95-99% median accuracy recovery can be targeted because QAD will recover more accuracy. This confirms that the quantization has been pushed far enough to bank the size and latency gains, while leaving clear room for QAD to close the gap in the next stage.
We expect a small but meaningful drop on the evaluation benchmarks with W4A16 but plan to use QAD to recover the drop. This confirms that the quantization has been pushed far enough to bank the size and latency gains, while leaving clear room for QAD to close the gap in the next stage.
Step 2: Quantization-aware distillation
During QAD, every forward pass of the student model runs through simulated quantization so the model can account for the quantization noise it will encounter at inference. At the same time, it is trained to match the teacher through a distillation loss. Through the teacher signal, the student is learning to reproduce the full behavior of the model it came from rather than just predicting the next token. Training against both signals together allows QAD to maintain high quality with aggressive quantization.
To learn more about the QAD process, see the end-to-end examples of QAD on NVIDIA Model Optimizer.
How to develop Nemotron 3.5 Lightning NVFP4 with QAD using Model Optimizer #
The following sections explain the process we used to develop the Nemotron 3.5 Lightning NVFP4 checkpoint using QAD with NVIDIA Model Optimizer.
Step 1: Obtain a PTQ checkpoint
The base model, NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, is the teacher. To create the student, run PTQ on the same base to quantize it to W4A16-NVFP4. Since both come from the same model, the student has a well-matched teacher to learn from.
We tried several PTQ recipes for the student, which differ in how the weights are calibrated and how aggressively the Mamba projections and KV cache are quantized. This choice carries into training: max-calibrated recipes feed dynamic scale QAD, while MSE-based recipes feed frozen-scale QAD (see Step 2 of this section).
A few settings are shared across every recipe. All of them quantize the lm_head to W4A16, a choice we call a faithful lm_head, while attention projection layers stay in BF16. Calibration uses 1,000 samples and runs on a single NVIDIA DGX B300. The last recipe, four_over_six plus NVFP4 KV, is the most aggressive. It pushes only K and V to NVFP4 (W4A4) and leaves the (QK^{\mathsf{T}}) and (\mathrm{attn} \cdot V) batched matrix multiplications in BF16, with Q left unquantized.
It’s advantageous to quantize more aggressively at the PTQ stage than if PTQ were the final step, because QAD recovers accuracy afterward. This enables you to select settings a PTQ-only recipe would avoid, such as taking the Mamba linear layers to W4A16 instead of the safer FP8.
In fact, we want the PTQ checkpoint to show a small but meaningful drop on the evaluation benchmarks, bringing median accuracy recovery between 95 to 99%. This drop confirms we pushed hard enough to bank the size and latency gains, while leaving clear room for QAD to close the gap in the next stage.
We evaluated many PTQ recipes, focusing on the five recipes below. For each recipe, we performed PTQ calibration with sequence length ranging from 8k to 128k. Prior experiments suggest longer sequence length produces better PTQ results. We found that a 32K sequence length provided the best PTQ results on four_over_six and performed all QAD experiments on the 32K calibration four_over_six checkpoint. After evaluating different PTQ recipes, we found the four_over_six with W4A16 Mamba linears provided the best tradeoff of accuracy degradation to boost in inference performance. For full accuracy details, see the QAD Checkpoint evaluations section.
Table 1 shows the five PTQ recipes used to build the Nemotron 3.5 Lightning student checkpoints. Each recipe is defined by its weight format for the MoE, shared, and lm_head layers, its calibration method, its Mamba in/out projection format, and its KV cache format. All five use W4A16 NVFP4 weights and range from max-calibrated dynamic recipes to MSE-based static recipes, with the most aggressive variant pushing the KV cache to NVFP4. Light Green rows use dynamic scaling and light gray rows use static scaling in the training step.
| Recipe | MoE / shared / lm_head weights | Calibration | Mamba in/out_proj | KV cache |
|---|---|---|---|---|
| max | W4A16 dynamic NVFP4 | max | W4A16 NVFP4 | FP8 |
| mamba_fp8_max | W4A16 dynamic NVFP4 | max | FP8 (W+A) | FP8 |
| MSE | W4A16 static NVFP4 | MSE (mean squared error) | W4A16 NVFP4 | FP8 |
| four_over_six | W4A16 static NVFP4 | 4/6 (MSE over M=6 versus M=4, | ||
static NVFP4NVFP4
Table 1. The five PTQ recipes used to build the Nemotron 3.5 Lightning student checkpointsYou can reproduce this recipe on your own model using NVIDIA Model Optimizer. The Hugging Face PTQ example walks through quantizing a Hugging Face model to NVFP4 using the different PTQ recipes previously explained.
import modelopt.torch.quantization as mtq
def forward_loop(model):
for batch in calib_data:
model(batch)
model = mtq.quantize(model, mtq.W4A16_NVFP4_CFG, forward_loop=forward_loop)
The same Hugging Face PTQ example also covers the calibration data, supported formats, and export options. Once you have a PTQ checkpoint, you can move on to the QAD training configuration and scale strategy previously described.
Step 2: QAD training
This section explains the QAD training, including configuration and quantization scale handling.
Setting the training configuration
With the student checkpoint in hand, the next step is deciding how to train it. Two choices mattered most in our ablations: the sequence length for training and the data for distillation.
Sequence length: Sequence length turned out to be critical for certain benchmarks, especially the longer-context ones, where training too short leaves accuracy on the table. Post-training supervised fine-tuning (SFT) used roughly 522K tokens, and our ablations showed that 522K sequence length was necessary for preserving long-context performance. To balance compute resources with sequence length, we performed initial ablations with 256K sequence length, and then scaled up to 522K for the final QAD run.
Datasets: For the data mix, we ablated a few internal mixes before landing on our final recipe. For anyone looking to reproduce this work, we recommend starting from the open datasets NVIDIA has released, Nemotron-Post-Training v1 and Nemotron-Post-Training v2, which cover a similar distribution to what we used.
Distillation recipe: The student starts from the PTQ checkpoint: the BF16 model is quantized first to NVFP4 using the Nemotron-3.5-Lightning-30B-A3B/lightning quantization recipe. Then distill from there rather than training the quantized weights from scratch. On each step, the same batch runs through both the BF16 teacher and the NVFP4 student, and the student is trained to match the teacher through a KL divergence distillation loss on the logits, with the teacher kept frozen as the reference signal. Distill at a constant learning rate of 5e-6 with no warmup, dropout disabled, and gradient clipping at 1.0, across two nodes times 8 GPUs (TP=2, EP=4). This is the same QAD workflow that ships with the NVIDIA Model Optimizer.
Quantization scale handling during QAD
With the checkpoint and training configuration set, the last choice is how to handle the quantization scales during QAD. We tried two strategies, and the application depends on how the PTQ checkpoint from Step 1 was calibrated.
Figure 2 shows dynamic versus frozen scale, in which both lanes share the same quantized forward pass consisting of activations, simulated quantization, GEMM, loss with gradients returning to the FP weights. The top lane recomputes its quantization scale each step from the current tensors; the bottom lane keeps the PTQ-calibrated scale locked and updates only the weights. W(t) is the BF16 weights at training step t, s(t) is the scale recomputed from those weights at that same step, and s* is that scale captured once and held for the whole run.
Dynamic scale QAD: Starts from a max calibrated PTQ checkpoint. This is either the max
or mamba_fp8_max
quantization config from Table 1. Both carry dynamic weight and activation scales, so the scales are recomputed on the fly during training and both the weights and the scales adapt as the model learns.
Frozen scale QAD: Starts from an MSE based PTQ checkpoint, such as MSE
, four_over_six
, or four_over_six + NVFP4 KV
from Table 1. MSE and four over six arrive at their scales through a search to minimize quantization error, which is far too expensive to repeat at every step. Instead of recomputing the scales, take the scales found during PTQ and freeze them during training. Only the weights are updated, while the scales stay fixed at their calibrated values.
The scale strategy is chosen before training time. It follows directly from the PTQ recipe selected in Step 1. Max-calibrated checkpoints lead to dynamic scale QAD, and MSE-based checkpoints lead to frozen scale QAD. It can be a good idea to try multiple PTQ recipes with a mix of dynamic and frozen scales, and evaluate them to assess which one will benefit from QAD.
QAD checkpoint evaluations #
The following experiments on intermediate Lightning checkpoints demonstrate how QAD allows for more aggressive quantization than PTQ alone. Both the PTQ and QAD checkpoints use the same aggressive W4A16 quantization format recipe and the same 21.19 GB footprint, against 65.85 GB for the BF16 baseline, so any difference between them comes from the method rather than from extra memory. QAD runs on intermediate checkpoints A and B were performed on 256K sequence length for rapid experimentation, whereas the final checkpoint QAD used 524K sequence length.
Checkpoint A
Checkpoint A (Table 2) is an SFT intermediate checkpoint. Checkpoint B (Table 3) is an intermediate RL checkpoint. We evaluated these checkpoints and found that QAD can recover some accuracy loss from aggressive quantization on only PTQ.
| Benchmark | BF1665.85 GB | Aggressive PTQ21.19 GB | QAD21.19 GB | QAD gainvs PTQ |
|---|---|---|---|---|
| MMLU-Pro | 81.27 | 80.10 | 81.04 | +0.94 |
| GPQA-D | 77.08 | 75.76 | 77.34 | +1.58 |
| AIME 2025 | 86.72 | 83.02 | 86.15 | +3.13 |
| AIME 2026 | 87.81 | 86.46 | 87.24 | +0.78 |
| SciCode Subtask | 35.21 | 32.36 | 35.72 | +3.37 |
| SciCode Problem | 13.28 | 10.63 | 12.19 | +1.57 |
| AA-LCR | 54.00 | 52.38 | 53.37 | +0.99 |
| AA-Omni Acc. | 14.05 | 13.62 | 14.57 | +0.95 |
| IFBench | 74.00 | 73.00 | 72.96 | -0.04 |
| HLE | 12.14 | 9.78 | 12.33 | +2.55 |
| LM Arena Proxy | 10.05 | 8.81 | 10.05 | +1.23 |
| Median score recovery | 100.00% | 96.33% | 99.72% | +3.39 |
Table 2. Checkpoint A: Aggressive quantization with PTQ introduces accuracy loss. QAD recovers most of it by distilling the full-precision model into the quantized one. PTQ reaches 96.33% median score recovery, while QAD reaches 99.72% and improves 10 of the 11 benchmarks
We tried QAD first on an early pre-release Checkpoint A, quantized more aggressively than what eventually shipped: W4A16 pushed all the way through the Mamba linear projections. Post-training quantization alone landed at 96.33% median accuracy recovery, with the largest regressions falling on the reasoning and coding benchmarks, where AIME 2025 dropped 3.70 points, SciCode Subtask 2.85, and HLE 2.36. Two hundred iterations of QAD from that same checkpoint brought recovery to 99.72%, returning AIME 2025 to within 0.57 points of BF16 and placing both SciCode Subtask and HLE slightly above it.
Checkpoint B
Table 3 shows checkpoint B, scored on the updated evaluation suite, with the same result. PTQ reaches 95.84% median score recovery and QAD reaches 98.53%, again at an identical 21.19 GB. The AA v4.1 Index is the clearest single measure here, rising 3.45 points from 20.03 to 23.48 against a BF16 baseline of 24.81. QAD improves 5 of the 9 benchmarks, so the median rather than any individual score carries the result.
| Benchmark | BF1665.85 GB | Aggressive PTQ21.19 GB | QAD21.19 GB | QAD gainvs PTQ |
|---|---|---|---|---|
| AA v4.1 Index | 24.81 | 20.03 | 23.48 | +3.45 |
| GPQA-D | 77.37 | 75.19 | 77.46 | +2.27 |
| HLE | 10.84 | 10.89 | 11.35 | +0.46 |
| AA-LCR | 51.00 | 47.38 | 50.25 | +2.88 |
| AA-Omni Acc. | 17.58 | 16.78 | 16.60 | -0.18 |
| AA-Omni Non-Halluci. | 62.24 | 69.02 | 64.67 | -4.35 |
| SciCode Subtask | 41.12 | 39.57 | 38.98 | -0.59 |
| Tau3 Banking | 8.04 | 5.77 | 8.25 | +2.48 |
| GDPval Norm. Elo | 18.60 | 16.46 | 12.90 | -3.56 |
| Median score recovery | 100.00% | 95.84% | 98.53% | +2.69 |
Table 3. Checkpoint B, scored on the updated evaluation suite, shows the same result as Checkpoint A
We repeated the experiment on a later Checkpoint B and observed the same pattern: 95.84% to 98.53% median recovery. We also measured the AA Index, which rose 3.4 points from 20.03 to 23.48 against a BF16 baseline of 24.81. The launch NVFP4 recipe is deliberately more conservative, so PTQ already sits close to baseline and QAD has considerably less to recover, with the remaining gains narrowing to a subset of the agentic benchmarks. Overall, QAD recovers the accuracy loss that aggressive quantization introduces. PTQ fixes the weights to a low-precision grid and accepts whatever error that produces, while QAD allows the model to continue training against that error and adapt to it, so most of the degradation is recovered rather than absorbed.
That difference determines how far a recipe can be pushed. Configurations that PTQ leaves several points below baseline become viable under QAD, which means the memory and throughput budgets that would otherwise force a more conservative precision are now within reach.
Final NVFP4 checkpoint
Table 4 shows accuracy evaluations for the final NVFP4 checkpoint BF16 versus PTQ versus QAD. The final checkpoint was quantized conservatively to optimize accuracy, so PTQ already sits close to BF16 and there is little loss left for QAD to recover. Median score recovery is 99.24% for PTQ and 98.97% for QAD. The gains show up on the agentic and coding benchmarks instead, where QAD beats PTQ by 3.79 points on Terminal-Bench v2.1, 1.07 on SWE-Bench Multilingual, 0.65 on HLE and smaller margins on BrowseComp, SWE-Bench Verified, and PinchBench.
| Benchmark | BF16 | PTQ | QAD |
|---|---|---|---|
| AGGREGATE | |||
| Score recovery (median) | 100.00 | 99.24 | 98.97 |
| AA v4.1 Index | 24.51 | 24.05 | 23.94 |
| AGENTIC & CODING | |||
| Terminal-Bench v2.1 | 24.44 | 22.05 | 25.84 |
| SWE-Bench Multilingual | 37.13 | 37.00 | 38.07 |
| BrowseComp | 39.50 | 39.50 | 40.00 |
| SWE-Bench Verified | 52.20 | 52.20 | 52.60 |
| PinchBench | 83.86 | 84.78 | 85.15 |
| τ²-bench Telecom | 59.65 | 60.96 | 59.87 |
| τ³-bench Banking | 9.90 | 9.48 | 7.84 |
| KNOWLEDGE, REASONING & INSTRUCTION FOLLOWING | |||
| LM Arena proxy | 12.06 | 11.10 | 11.91 |
| HLE (w/o tools) | 10.80 | 10.70 | 11.35 |
| AA-Omniscience, accuracy | 17.47 | 16.35 | 16.80 |
| AA-Omniscience, non-halluc. | 68.88 | 69.20 | 69.03 |
| SciCode (subtask) | 32.51 | 31.21 | 30.73 |
| GPQA Diamond (w/o tools) | 76.45 | 75.38 | 74.43 |
| GDPval (norm. Elo) | 17.19 | 19.09 | 17.02 |
| IFBench (loose) | 72.00 | 74.04 | 71.21 |
| LONG CONTEXT | |||
| AA-LCR | 52.38 | 50.00 | 48.50 |
Table 4. Accuracy evaluations for the final NVFP4 checkpoint BF16 versus PTQ versus QAD
For this final checkpoint, we optimized for accuracy, meaning more conservative quantization using Nemotron-3.5-Lightning-30B-A3B/lightning_w4a16_nvfp4_4o6. PTQ therefore lands close to the BF16 baseline and QAD has much less loss to recover, with median score recovery at 99.24% and 98.97%, respectively.
The gains from QAD are smaller than under aggressive PTQ, but they show up where it matters: QAD recovers accuracy on several key agentic benchmarks, including Terminal-Bench v2.1, SWE-Bench Multilingual, BrowseComp, PinchBench, HLE, and AA-Omniscience accuracy.
How to run the QAD recipe with Model Optimizer #
The full QAD recipe ships in NVIDIA Model Optimizer as a single launcher megatron_lm_qad.yaml that runs the whole pipeline end-to-end. The same file works for both Nemotron 3 Nano and Nemotron 3.5 Lightning. Point it at the model you want and launch. The final NVFP4 checkpoint is published on Hugging Face.
source .env-slurm
uv run launch.py --yaml examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/megatron_lm_qad.yaml --yes
**Step 1: Create the PTQ student **
The first step here is to use PTQ to quantize the BF16 teacher into the NVFP4 student. Nemotron 3.5 Lightning does not use RoPE, so --max-position-embeddings
** has no effect on positional encoding here. Here we keep it equal to its maximum context length (1M) for consistency. For models that do use RoPE, leave it at the **config.json
value.
task_1: # quantize the BF16 teacher into the NVFP4 student
script: common/megatron_lm/quantize/quantize.sh
args:
- --seq-length 32768 --max-position-embeddings 1048576
- --calib-size 32
environment:
- QUANT_CFG: MAMBA_MOE_NVFP4_CONSERVATIVE_CFG
- TP: "1"
- EP: "4"
Step 2: Distill the student from the frozen teacher
QAD is enabled by setting the teacher through --export-kd-teacher-load
with --modelopt-enabled
added. Under the hood, the launcher invokes the Megatron-LM training entry point, finetune.sh, which runs the actual distillation loop against the frozen teacher.
To reproduce our setup, point the launcher at your PTQ student checkpoint and teacher, adjust the sequence length and data mix to match your target, and launch. We distill at a constant 5e-6 learning rate with dropout off and gradient clipping at 1.0, on a single Nemotron-Post-Training v2 chat shard:
task_2: # distill the NVFP4 student against the BF16 teacher
script: common/megatron_lm/train/sft.sh
args:
- --seq-length 32768 --max-position-embeddings 1048576
- --micro-batch-size 1 --global-batch-size 16
- --train-samples 6400 # -> 400 iterations
- --modelopt-enabled
- --export-kd-teacher-load /cicd/megatron-lm-bf16/.../BF16-MCore
- --lr 5.0e-6 --lr-decay-style constant --lr-warmup-samples 0
- --clip-grad 1.0 --weight-decay 0.0
- --attention-dropout 0.0 --hidden-dropout 0.0
environment:
- DATASET: nvidia/Nemotron-Post-Training-Dataset-v2
- TP: "1"
- EP: "4"
A final export task converts the trained student into a deployable Hugging Face NVFP4 checkpoint.
To reproduce on your own model, swap the model paths and choose your QUANT_CFG (which sets the matching scale strategy from Step 3); to train at a higher sequence length, set both –seq-length and –max-position-embeddings to a higher value, such as 524,288 (524k). For the complete launcher, see megatron_lm_qad.yaml.
How to run the QAD recipe with Megatron-Bridge
The same steps also run through Megatron-Bridge, a PyTorch-native library within the NeMo Framework that provides pretraining, SFT, and LoRA for popular language, vision-language, audio, and multimodal model support. It serves as a bridge between Hugging Face and Megatron Core providing bidirectional checkpoint conversion between the two formats. Projects can leverage Megatron Core parallelism capabilities or export models for various inference engines, with built-in verification.
The mbridge_qad.yaml launcher runs the entire process end-to-end as four tasks: tokenize the training data, quantize the BF16 model into the NVFP4 student, distill that student against the frozen teacher, and export a deployable Hugging Face checkpoint. Here we use pre-tokenized data, the Nemotron-Post-Training-Dataset-v2 chat split is tokenized once up front with megatron_preprocess_data
and passed to distillation through --data_paths
.
source .env-slurm
uv run launch.py --yaml examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/mbridge_qad.yaml --yes
This runs on eight nodes with four GPUs (TP=1, PP=1, CP=4, EP=16), giving DP=8, with microbatch size=1 and global batch size=64; 200 iterations cover roughly 419M training tokens.
The pipeline comprises the three steps detailed in this post, wrapped in four tasks:
- Import the BF16 model as the Megatron-Core teacher
- Quantize a separate student
- Distill the student against the frozen teacher
- Export the result
This runs on two nodes with eight GPUs (TP=2, PP=1, EP=4), giving DP=8, with microbatch size=1 and global batch size=16; train samples=6,400 yields 400 iterations.
Learn more #
QAD makes it possible to quantize aggressively and still hold accuracy close to the original model. On a compact, sparsely activated model such as Nemotron 3.5 Lightning, this is exactly what unlocks a strong NVFP4 checkpoint while preserving accuracy.
The full QAD recipe for Nemotron 3.5 Lightning is available in NVIDIA Model Optimizer. We hope you try it on your own models and build on what we have shared here.
To learn more, check out the following resources:
Nemotron 3.5 Lightning Model Optimizer 4o6 yaml recipeNemotron 3.5 Lightning Megatron LM QADNemotron 3.5 Lightning Megatron Bridge QAD yamlNVIDIA Model Optimizer Github RepoNVIDIA Model Optimizer QAD DocumentationNVIDIA Model Optimizer LLM PTQ Documentation
Acknowledgments
We would like to thank the Nemotron team for building and open sourcing Nemotron 3.5 Lightning and the post-training datasets that made this work possible, and the NVIDIA Model Optimizer team for the quantization and distillation tooling that powers this recipe.
We are grateful to Asma Kuriparambil Thekkumpate, Carlo del Mundo, Chenjie Luo, Daniel Lo, Daria Levy, Frank Sun, Hung-Yueh Chiang, James Shen, Jinhang Choi, Sweta Priyadarshi, Konstantinos Krommydas, Meng Xin, Rohan Joshi, Wei-Ming Chen, Victor Cui, Trenton Starkey, and Yaniv Galronfor their contributions to this work, including the recipe development, ablations, evaluations, and reviews that shaped the results presented here.