How to Run Nemotron 3.5 Super VL Locally: You Need 128GB, Not a Blackwell GPU

Can you run Nemotron 3.5 Super VL locally without a Blackwell GPU? Yes, if you have roughly 128 GB of memory. The GPU generation matters much less than that number.

NVIDIA released the model on October 15: a 120B vision-language MoE with 12B active parameters. The headline checkpoint is NVFP4, a 4-bit format that only Blackwell executes natively, and the model card lists four deployment targets: DGX Spark, one H200, two H100s, or one GB200. A day later there are GGUF, MLX, FP8 and EXL3 conversions on Hugging Face, and going by Unsloth's numbers, llama.cpp should run it with vision on a 128 GB Mac or a 96 GB workstation card. The RTX 5090 is the one popular card that gets left out.

I haven't run it. Everything below comes from the model card, the conversion repos and the Unsloth docs, with links throughout.

From the card:

off, on (medium) and on, set through the chat template I went looking for a separate text-only "Nemotron 3.5 Super" and there isn't one. The VL model is the 3.5 Super, and the card's benchmark section calls it that. The backbone was "initialized from Nemotron 3 Super Base", then distilled for 1T tokens from Nemotron 3 Ultra Base, extended to 1M context, given 67B tokens of software-engineering mid-training, and only then trained on vision. So the real predecessor is Nemotron 3 Super, the text-only model from March 11, 2026.

Both cards include NVFP4 scores, so this compares NVFP4 to NVFP4. Nemotron 3 Super was scored with NeMo Evaluator/NeMo Skills and 3.5 with NeMo Gym, though, so I'd treat the deltas as directional.

On reasoning and agentic tasks it's a big jump. Multilingual knowledge (MMLU-ProX) and instruction following went down, and NVIDIA doesn't comment on either.

On the vision side (NVFP4): MathVision 77.79, CharXiv reasoning 79.88, V* 85.95, OCRBench v2 EN 68.34 / ZH 59.03, MMLongBench-Doc 62.82, OSWorld-Verified 52.3, Video-MME v2 32.54. For coding it posts SWE-bench Verified 71.07, SWE-bench Pro 51.30 and Terminal-Bench 2.1 62.78.

NVFP4 isn't 4 bits across the board. The vision encoder, connector, Mamba convolutions and embeddings stay in BF16, and the MoE router stays in FP32. The card says inference "uses W4A4 on Blackwell and W4A16 on non-Blackwell platforms". In other words, a Hopper card still loads the small checkpoint but runs its matmuls with 16-bit activations. You get the memory savings without the FP4 tensor-core speed.

The repo is 75.1 GB across 88 safetensors shards. The Hugging Face sidebar says "66B params", which I read as the U8-packed 4-bit weights being counted per byte, not as a different model.

NVIDIA ran BF16 and NVFP4 with identical harnesses, and most rows are within noise. A few are not:

If you're building a browser agent, an 11-point drop on WebVoyager is a real problem, and the card doesn't mention it anywhere outside the table.

The checkpoint and quant sizes below come from the NVIDIA, ggml-org, Unsloth and rapid-mlx repos. The memory estimates are each source's own planning figures; I haven't measured any of them.

So a single consumer card can't hold the 120B at any bit width. NVIDIA's answer is Nemotron 3.5 Bolt 74B-A9B 24GB, compressed from Super VL with native 2-bit quantization, aimed at llama.cpp on 32 GB+ discrete GPUs (Ada Lovelace and Blackwell). If you own a 5090, I'd start there before trying CPU offload on the 120B.

For Mac owners, the rapid-mlx card reports a measured peak of 72.7 GB at 1K context on an M3 Ultra. It also says 96 GB Macs "are not yet verified" and that image input and MTP are not enabled. If you want vision on a Mac today, use llama.cpp with the mmproj file.

On DGX Spark, this is the card's own Quick Start, using vllm/vllm-openai:v0.32.0 (or the pinned nightly):

export MODEL=nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4
export DRAFT_MODEL=nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4-LiLiCorr

vllm serve $MODEL \
  --served-model-name nemotron-3.5-super-vl \
  --max-num-seqs 1 \
  --trust-remote-code \
  --gpu-memory-utilization 0.75 \
  --max-num-batched-tokens 8192 \
  --speculative_config.model $DRAFT_MODEL\
  --speculative_config.num_speculative_tokens 7 \
  --speculative_config.method dflash \
  --mamba-backend flashinfer \
  --mamba-ssm-cache-dtype float16 \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5 \
  --enable-mamba-shared-prefix-checkpoint \
  --video-pruning-rate 0.5 \
  --media-io-kwargs '{"video": {"fps": 2, "num_frames": 256}}' \
  --block-size 64 \
  --prefix-match-unit 64 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3

If you copy the command from the card instead of from here, check the num_speculative_tokens 7 \ line. On the card it has a tab after the backslash, so bash ends the command there and then tries to run --speculative_config.method as its own command.

One more inconsistency: the card's own deployment table says DGX Spark uses "MTP with 5 draft tokens", but the command above uses LiLiCorr with 7. The card recommends LiLiCorr on Spark "in all scenarios", so I'd go with the command.

Everywhere else, use llama.cpp. Unsloth's recipe after building with -DGGML_CUDA=ON (or OFF on Mac, where Metal is the default):

hf download unsloth/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-GGUF \
    --local-dir models/nemotron-super-vl \
    --include "*UD-Q4_K_XL*.gguf" "*mmproj*.gguf"

./llama.cpp/llama-server --model "$MODEL_FILE" --mmproj "$MMPROJ_FILE" \
    --alias nemotron-super-vl --ctx-size 16384 --parallel 1 \
    --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0.0 \
    --media-path "$PWD/media" --host 127.0.0.1 --port 8001

Start at 16K context, not 1M. KV cache, video frames and the 1.73 GB BF16 mmproj all come out of the same memory budget. For video you need ffmpeg and ffprobe on your PATH.

Nemotron 3 Super shipped under the NVIDIA Nemotron Open Model License. Nemotron 3.5 Super VL ships under OpenMDW 1.1, which grants rights "to deal in the Model Materials without restriction". There are no field-of-use clauses and no restrictions on outputs. You have to keep the notices when you redistribute, and your rights end if you sue claiming the model infringes a patent or copyright. That's closer to MIT than to most "open" model licenses, and it matters more to me than the FP4 kernels do.

NVFP4 is a good format attached to a sales pitch. The quality loss is small on most rows and large on a couple of agentic ones, and the speed advantage only exists on Blackwell, while the community had working GGUF and MLX builds within a day. If you already own a DGX Spark, the official path is the most polished option. If you don't, spend your money on memory, not on FP4 tensor cores.

So, would you buy Blackwell hardware just to run NVFP4 models natively, or is a 128 GB Mac plus a GGUF good enough for you? And if you've run the 120B with offload on a 5090, post your tokens per second. I'd like to see them.