Nvidia Nemotron 3.5 Super and Lightning

NVIDIA released Nemotron 3.5 Super, a 120-billion-parameter, 12-billion-active Mixture-of-Experts Hybrid Mamba-Transformer multimodal model, and Nemotron 3.5 Lightning, a 30-billion-parameter, 3-billion-active text model, both trained for context lengths up to 1 million tokens. Nemotron 3.5 Super scores 51.78% on SWE-bench Pro, surpassing Nemotron 3 Ultra with roughly 4.6× fewer active parameters, and gains up to 9.82 percentage points across five Indic benchmarks after a 100-billion-token Indic continued-pretraining stage. NVIDIA released BF16 and NVFP4 checkpoints with optimized inference recipes and standalone speculative drafters.

NVIDIA Nemotron 3.5 Super and Lightning Published: Technical Report https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3.5-Super-and-Lightning-Technical-Report.pdf We introduce NVIDIA Nemotron 3.5 Super and NVIDIA Nemotron 3.5 Lightning , open and efficient Mixture-of-Experts Hybrid Mamba-Transformer models designed for reasoning, long-context understanding, and agentic workloads. Nemotron 3.5 Super is a 120-billion-parameter, 12-billion-active multimodal model , while Nemotron 3.5 Lightning is a smaller 30-billion-parameter, 3-billion-active text model . Both are designed for efficient deployment and combine large-scale distillation, long-context training, native speculative decoding, and efficiency-focused quantization. Super further supports reasoning across text, images, video, and interactive tool environments. Key Features 1. Efficient hybrid architecture: Mixture-of-Experts Hybrid Mamba-Transformer models activate only a fraction of their total parameters for each token. 2. Native speculative decoding: Multi-Token Prediction MTP heads accelerate generation, with dedicated post-training that improves acceptance at deeper draft positions. 3. Long-context support: Both models are trained for context lengths up to 1 million tokens . 4. Agentic and multimodal capabilities: Super supports text, images, video, software-engineering agents, visual tool use, and long-running interactive workloads. 5. Reasoning control: Super supports reasoning-off, regular, and medium-effort modes, as well as inference-time reasoning-budget control. 6. Efficient deployment: Released BF16 and NVFP4 checkpoints are accompanied by optimized inference recipes and standalone speculative drafters. Training Nemotron 3.5 Both models begin with a dedicated 100-billion-token MTP warm-up , followed by 1 trillion tokens of distillation from the next-larger Nemotron 3 model. Nemotron 3.5 Super is distilled from Nemotron 3 Ultra Base, while Lightning is distilled from Nemotron 3 Super Base. Following distillation, Lightning undergoes a 21-billion-token long-context phase that extends it to a 1-million-token context length. Super proceeds through the additional continued-pretraining and mid-training stages below. Nemotron 3.5 Super then undergoes additional training targeted at multilingual, long-context, and agentic use cases: - Indic continued pretraining: A 100-billion-token stage includes synthetic and natural Indic data spanning 22 Indian languages and 34 language-script configurations. It improves all five evaluated Indic benchmarks, with gains of up to 9.82 percentage points . - Long-context training: A 35-billion-token continued-pretraining stage extends Super to a 1-million-token context length . - Agentic software-engineering mid-training: A 65.4-billion-token stage at 256K context uses repositories, issue–pull-request pairs, commit histories, build and CI logs, and agent trajectories. It improves long-horizon software engineering and robustness across agent harnesses. - Long-context recovery: Because Super’s 256K mid-training stage reduced its RULER-1M score, a final 9-billion-token recovery stage at 1-million-token context restores the score from 33.5 to 81.0. Super’s post-training pipeline combines supervised fine-tuning, Reinforcement Learning from Verifiable Rewards RLVR , Reinforcement Learning from Human Feedback RLHF , specialized teacher training, and Multi-Teacher On-Policy Distillation MOPD . This process transforms the text-only base model into a multimodal agent while consolidating capabilities from reasoning, agentic, alignment, video, and vision-tool-use teachers. Lightning follows a lighter text-only pipeline using supervised fine-tuning and RLVR. Key Highlights - Nemotron 3.5 Super achieves 51.78% on SWE-bench Pro , surpassing Nemotron 3 Ultra while using approximately 4.6× fewer active parameters . - Agentic mid-training improves DeepSWE pass@1 from 6.4% to 10.4% and improves robustness across multiple software-engineering agent harnesses. - Parallel agent execution followed by self-verification improves Terminal-Bench 2.1 from 62.7% to 71.9% , DeepSWE from 24.6% to 34.5% , and BrowseComp from 71.5% to 79.8% on the evaluated setups. - MTP boosting improves average speculative-decoding acceptance length from 4.287 to 4.718 under greedy decoding and from 3.787 to 4.612 under probabilistic draft sampling. - The released Nemotron 3.5 Super NVFP4 checkpoint achieves 99.62% median accuracy recovery relative to BF16 across 36 benchmarks. - On a single NVIDIA GB200 GPU, Nemotron 3.5 Super reaches up to 4,856 output tokens/s/GPU while sustaining at least 50 tokens/s/user on the evaluated 20-turn agentic workload. - Under the evaluated interactive serving conditions, NVFP4 delivers 2.43× the throughput of BF16 . Nemotron 3.5 Super delivers 2.34× and 1.90× the throughput of Qwen3.5 122B-A10B and Qwen3.8 Flash-Next, respectively. Efficient Inference Nemotron 3.5 is optimized for interactive, long-context agentic workloads. Hybrid-state caching, prefix reuse, Mamba checkpointing, GPU-kernel selection, and speculative decoding are jointly tuned to sustain responsive per-user generation rates at high aggregate throughput. Post-training MTP boosting improves native speculative decoding, while standalone parallel drafters target low-concurrency, memory-bound deployments: - LiLiCorr for Nemotron 3.5 Super - DFlash and DSpark for Nemotron 3.5 Lightning Nemotron 3.5 Super is validated across NVIDIA A100, H100, B200, GB200, and GB300 platforms, as well as DGX Spark, DGX Station, and RTX-class GPUs. Its quantization and speculative-decoding configurations can be tuned to the target platform and workload. Open Source We are releasing base, post-trained BF16, and NVFP4-quantized checkpoints, together with speculative drafters, datasets, model cards, cookbooks, evaluation recipes, and training recipes. Checkpoints - Nemotron 3.5 Super VL 120B-A12B Base BF16 https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Base-BF16 — base model - Nemotron 3.5 Super VL 120B-A12B BF16 https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 — post-trained model - Nemotron 3.5 Super VL 120B-A12B NVFP4 https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 — post-trained and NVFP4-quantized model - Nemotron 3.5 Super VL 120B-A12B NVFP4 LiLiCorr https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4-LiLiCorr — standalone LiLiCorr drafter - Nemotron 3.5 Lightning 30B-A3B Base BF16 https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Base-BF16 — base model - Nemotron 3.5 Lightning 30B-A3B BF16 https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 — post-trained model - Nemotron 3.5 Lightning 30B-A3B NVFP4 https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 — post-trained and NVFP4-quantized model - Nemotron 3.5 Lightning 30B-A3B NVFP4 DFlash https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash — standalone DFlash drafter - Nemotron 3.5 Lightning 30B-A3B NVFP4 DSpark https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark — standalone DSpark drafter Data and Recipes - Nemotron-Pretraining-Indic-v1: approximately 2.83 trillion synthetic output tokens across 22 Indian languages and 34 language-script configurations. - Post-training datasets and environments: resources for supervised fine-tuning, reinforcement learning, reasoning, coding, software engineering, tool use, and multilingual capabilities. - Nemotron 3.5 Super evaluation recipes https://github.com/NVIDIA-NeMo/Gym/blob/main/nemotron recipes/super-3.5/reproducibility.md - Nemotron 3.5 Lightning evaluation recipes https://github.com/NVIDIA-NeMo/Gym/blob/main/nemotron recipes/lightning-3.5/reproducibility.md - NVIDIA Nemotron Developer Repository https://github.com/NVIDIA-NeMo/Nemotron Technical Report The Nemotron 3.5 Super and Lightning Technical Report https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3.5-Super-and-Lightning-Technical-Report.pdf provides full details on pretraining, post-training, quantization, inference optimization, evaluation methodology, and benchmark results.