{"slug": "nvidia-nemotron-bolt-74b-2-bit-how-to-run-it-locally-on-24gb", "title": "NVIDIA Nemotron Bolt 74B 2-Bit: How to Run It Locally on 24GB", "summary": "NVIDIA released Nemotron 3.5 Bolt 74B-A9B, a mixture-of-experts model with 74 billion total parameters and roughly 9 billion active per token that fits in 24GB of VRAM because its expert layers are quantized to 2 bits, while its Mamba/attention layers run at NVFP4 and embeddings at 8-bit. NVIDIA derived the model from the 120-billion-parameter, 12B-active Nemotron 3.5 Super using a pruning, distillation and quantization-aware training pipeline plus an architecture search method called Star Elastic that cut active experts per token from 22 to 9, compressing the model from 148GB in BF16 to 22GB. On NVIDIA's own benchmarks the compression dropped scores only slightly, from around 90 to 88 on one open benchmark and from around 89 to 86 on GPQA Diamond, while real-world throughput on an RTX Spark reaches about 84 tokens per second, trailing Qwen 3 and Gemma 4 models of similar size.", "body_md": "# NVIDIA Nemotron Bolt 74B 2-Bit: How to Run It Locally on 24GB\n\nNVIDIA's Nemotron Bolt packs a 74B MoE model into 24GB using 2-bit experts. Here's how it works, how it performs, and how to run it.\n\n## What is Nemotron Bolt 74B-A9B and why does it matter?\n\nNVIDIA Nemotron 3.5 Bolt 74B-A9B is a mixture-of-experts model with 74 billion total parameters and roughly 9 billion active per token, built on NVIDIA’s hybrid Mamba-transformer architecture. It handles text, images, and video, supports context windows up to 256K tokens, and ships with its experts quantized to 2 bits, letting the entire model fit inside 24GB of VRAM. That makes it one of the first large, capable MoE models designed from the ground up to run on a single consumer-grade GPU like the RTX Spark, alongside other apps, rather than needing a multi-GPU server.\n\nThe model matters less because of raw benchmark scores and more because of how NVIDIA got there. It’s not a model someone rounded down after training. It’s the end product of a pruning, distillation, and quantization-aware training pipeline that started with a 120-billion-parameter parent model. That pipeline is arguably more useful to understand than the model itself, because it previews where small-but-capable local models are headed.\n\n## TL;DR\n\n- **Nemotron Bolt 74B-A9B** is a 74B-parameter MoE model with about 9B active parameters per token, quantized so the whole thing fits in 24GB of VRAM.\n- The 2-bit claim is specific to the **expert layers** ; the Mamba/attention layers run at NVFP4 and embeddings sit at 8-bit, making this a mixed-precision model rather than uniformly 2-bit.\n- NVIDIA used **quantization-aware training and distillation (QAD)** , not post-training rounding, which is why it avoids the steep quality collapse normally seen below 4-bit.\n- The model is **derived from Nemotron 3.5 Super** , a 120B-parameter, 12B-active parent, shrunk via an architecture search method called Star Elastic that cut active experts per token from 22 down to 9.\n- On NVIDIA’s own benchmarks, compressing from BF16 (148GB) to the 2-bit release (22GB) dropped scores only slightly, for example around 90 to 88 on one open benchmark and around 89 to 86 on GPQA Diamond.\n- Real-world throughput on an RTX Spark comes in around **84 tokens per second** , which trails Qwen 3 and Gemma 4 models of similar size, meaning Bolt is pitched as a smarter-per-gigabyte model, not a faster one.\n- The same recipe (prune, distill, quantization-aware train) is becoming a pattern across the industry, with Google’s Gemma 4, OpenAI’s GPT-OSS, and Moonshot’s Kimi models all applying some version of it.\n\n## How does 2-bit quantization usually break models, and how did NVIDIA avoid it?\n\nThe standard local-AI rule of thumb has been to pick the largest model you can fit at 4-bit and avoid going much lower. Post-training quantization, the kind used for most GGUF Q4 or Q8 files, takes a model trained in 16-bit and rounds every weight down afterward, usually guided by a small calibration dataset to minimize error. At 8-bit this is nearly free in terms of quality loss. At 4-bit it’s usually still close to free, which is why 4-bit became the default.\n\nBelow 4-bit, though, post-training quantization tends to fall apart. The transition between 2-bit and 3-bit is where researchers typically see the sharpest quality drop. At 3-bit and above, a quantized model usually stays reasonably close to the original. At 2-bit and below, the internal representations change so much that the model effectively needs to relearn how to function, not just get rounded.\n\nNVIDIA’s approach sidesteps this by not treating quantization as a final, separate step. It uses quantization-aware training (QAT), where low-bit rounding is simulated during training itself while a full-precision copy of the weights handles the actual updates. The model learns to operate well on the lower-bit “grid” instead of being forced onto it after the fact. NVIDIA adds a twist called quantization-aware distillation (QAD): the quantized model acts as the student, distilling from the full-precision version, which lets the team reuse the benefits of supervised fine-tuning and reinforcement learning without needing to rerun the full training dataset.\n\nThis pattern isn’t unique to NVIDIA. Google used QAT for Gemma 3 and extended it across the whole Gemma 4 family, pushing the smallest models’ token-generation layers down to 2-bit while keeping core reasoning layers at higher precision. OpenAI used a related approach for its GPT-OSS 120B model, which is part of why it fits on a single 80GB GPU. Moonshot has applied similar ideas to Kimi. The common thread: major labs are increasingly planning for low-bit deployment during training rather than leaving it to the open-source community to figure out afterward.\n\n## How was Nemotron Bolt actually built?\n\nThe pipeline runs in three stages.\n\n**Stage one: prune with a memory budget in mind.** NVIDIA starts from Nemotron 3.5 Super, a 120-billion-parameter model with about 12 billion active parameters. A method called Star Elastic performs an architecture search under a fixed memory budget: a smaller router decides, layer by layer, how many experts to keep and how wide each MoE block should be. The number of active experts per token drops from 22 to 9. Critically, the budget isn’t just “hit 74B parameters.” It’s defined as fitting under 24GB at batch size 1 with a 128K context window, and that budget already accounts for the experts eventually landing at 2-bit. The Mamba state layers stay at 16-bit throughout this stage. The pruned model is then distilled from its parent using around 120 billion tokens, mostly at 8K context for speed, with later training stretching to 128K to restore long-context performance.\n\n*demands*of enterprises.\n\n**Passed**\n\n*✓*SOC 2 Type 2\n\n*✓*HIPAA\n\n*✓*CAIQ attestation\n\n*✓*OCSF logging\n\n*✓*Role-based access\n\n*✓*Self-hosted\n\n**Stage two: standard post-training.** The vision encoder, separated earlier, gets reattached. NVIDIA runs further supervised fine-tuning, merges checkpoints, and does multiple rounds of reinforcement learning aimed at agentic tasks. At this point the model is still in BF16.\n\n**Stage three: quantization-aware distillation.** This is where the 2-bit experts, NVFP4 Mamba/attention layers, and 8-bit embeddings get baked in through QAD rather than a simple post-hoc rounding pass.\n\n## How fast is it, and is it actually worth running locally?\n\nSpeed is where the tradeoffs show up. On an RTX Spark, NVIDIA’s own numbers put Bolt at around 84 tokens per second, which is behind comparably sized Qwen 3 and Gemma 4 releases. Testing on an NVIDIA GB10-class system (DGX Spark hardware) showed throughput around 220 tokens per second, noticeably slower than running the full Nemotron 3.5 Super model on the same hardware, since Super has more headroom for parallel compute even though it’s a larger file. The takeaway: this is a smarter-per-gigabyte model, not a faster-per-gigabyte one.\n\nThree reasons still make the tradeoff reasonable for local setups. First, at batch size 1 (the typical local use case), you’re bottlenecked by memory bandwidth, not compute, so fewer bits per weight directly means fewer bytes to move and more tokens per second, all else equal. Second, headroom matters: an agent setup running a KV cache, embedding model, speech model, browser, and IDE all on one machine needs the LLM itself to leave room. Third, the choice between a 35B model at 4-bit and a 70B model at 2-bit in a similar footprint is a real design decision, and NVIDIA’s benchmark drops (roughly 90 to 88 on one benchmark, 89 to 86 on GPQA Diamond when moving from the 148GB BF16 version to the 22GB quantized version) suggest the bigger, more aggressively quantized model holds up better than expected.\n\n## What are the alternatives to this approach?\n\nTwo other techniques are worth knowing if you’re tracking this space. Expert pruning, used in methods like Cerebras’s REAP, drops entire experts from a MoE model rather than shrinking every layer uniformly. Research here shows pruning whole experts tends to outperform merging them, since merging damages the router’s ability to pick the right expert, though which experts get cut depends heavily on the calibration data used (a code-focused calibration set will preserve coding ability at the expense of other skills).\n\nThe more radical alternative is Microsoft’s BitNet, which trains models from scratch with weights restricted to -1, 0, or 1, the so-called 1.58-bit approach. It shows promise but currently lacks proper hardware acceleration on mainstream GPUs, which limits its practicality compared to NVIDIA’s train-big-then-compress strategy.\n\n## Frequently Asked Questions\n\n### Is Nemotron Bolt 74B really running entirely at 2-bit?\n\nNo. The 2-bit precision applies specifically to the mixture-of-experts layers. The Mamba/attention (member) layers run at NVFP4 (4-bit), and the embeddings and output layer run at 8-bit. It’s a mixed-precision model, and NVIDIA is explicit about this in the model card.\n\n### How much VRAM do I need to run Nemotron Bolt locally?\n\nThe quantized release is designed to fit in 24GB of VRAM, which is why NVIDIA paired it with the RTX Spark class of hardware. The full BF16 parent model, by contrast, runs around 148GB.\n\n*mission-critical*enterprise applications.\n\n**✓ HIPAA**\n\n**✓ SOC 2**\n\n**✓ Self-hosted**\n\n### Why does a 2-bit model actually perform well here, when most 2-bit quantization doesn’t?\n\nBecause it wasn’t quantized after training. NVIDIA used quantization-aware training and quantization-aware distillation, simulating low-bit rounding during training itself so the model adapts to the lower precision instead of being forced onto it afterward.\n\n### Is Nemotron Bolt faster than similarly sized local models?\n\nNot necessarily. NVIDIA’s own figures show around 84 tokens per second on the RTX Spark, which is slower than comparable Qwen 3 and Gemma 4 models. The appeal is capability per gigabyte of VRAM, not raw speed.\n\n### Where does Nemotron Bolt come from architecturally?\n\nIt’s derived from Nemotron 3.5 Super, a 120-billion-parameter model with 12 billion active parameters. NVIDIA pruned it using a method called Star Elastic under a strict memory budget, cut active experts per token from 22 to 9, then distilled and fine-tuned the result before applying quantization-aware distillation.", "url": "https://wpnews.pro/news/nvidia-nemotron-bolt-74b-2-bit-how-to-run-it-locally-on-24gb", "canonical_source": "https://www.mindstudio.ai/blog/nvidia-nemotron-bolt-74b-2bit-local/", "published_at": "2026-11-10 00:00:00+00:00", "updated_at": "2026-11-10 10:25:23.756961+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-infrastructure", "machine-learning"], "entities": ["NVIDIA", "Nemotron 3.5 Bolt 74B-A9B", "Nemotron 3.5 Super", "Star Elastic", "RTX Spark", "Qwen 3", "Gemma 4", "GPQA Diamond"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/nvidia-nemotron-bolt-74b-2-bit-how-to-run-it-locally-on-24gb", "markdown": "https://wpnews.pro/news/nvidia-nemotron-bolt-74b-2-bit-how-to-run-it-locally-on-24gb.md", "text": "https://wpnews.pro/news/nvidia-nemotron-bolt-74b-2-bit-how-to-run-it-locally-on-24gb.txt", "jsonld": "https://wpnews.pro/news/nvidia-nemotron-bolt-74b-2-bit-how-to-run-it-locally-on-24gb.jsonld"}}