Running GPT-OSS-120B on a Single NVIDIA DGX Spark - A Practical Guide NVIDIA's DGX Spark single-node AI appliance can run OpenAI's open-weight GPT-OSS-120B sparse MoE model at ~50 tokens/s using optimized engines like SGLang or llama.cpp, making it viable as a local coding backend for tools such as Claude Code. The 128 GB unified memory fits the MXFP4 quantized model with room for KV cache, while the low active-parameter count (5.1B per token) avoids bandwidth bottlenecks typical of dense models. Running GPT-OSS-120B on a Single NVIDIA DGX Spark - A Practical Guide Note on the model name:OpenAI’s open-weight family ships as gpt-oss-20b and gpt-oss-120b . There is no 130B variant — this guide targets, which is the one sized to fit the Spark’s unified memory. gpt-oss-120b A practical, single-node setup guide for serving gpt-oss-120b as a local coding backend on the GB10 Grace Blackwell DGX Spark, and wiring it into Claude Code. 1. Why this model fits the Spark The DGX Spark has 128 GB of coherent unified LPDDR5x ~119.7 GB addressable by the GPU but only ~273 GB/s of memory bandwidth . Token generation is bandwidth-bound, so bandwidth — not capacity — is the limiting factor. gpt-oss-120b is a good match for two reasons: It fits. In its native MXFP4 weight format the full model loads into the ~120 GB unified pool with room left for KV cache. It’s a sparse MoE. The model has ~117B total parameters but activates only ~5.1B per token. Generation speed scales with active parameters against bandwidth, so it runs far faster than a dense model of comparable footprint. For reference, on the same box a dense ~32B model is bandwidth-starved ~9–10 tok/s , while small-active MoE models run several times faster. Published gpt-oss-120b results on the Spark land around ~50 tokens/s on an optimized engine SGLang , which is usable for an interactive coding agent. Rule of thumb for the Spark:prefer MoE models with low active-parameter counts; avoid large dense models. 2. Prerequisites | Requirement | Detail | |---|---| | Hardware | NVIDIA DGX Spark GB10 , 128 GB unified memory | | OS | DGX OS Ubuntu-based, ARM64 / aarch64 | | GPU stack | CUDA + drivers preinstalled on DGX OS; Blackwell compute capability sm 121 | | Firmware | Update to a current firmware version before serving see §6 | | Disk | The 120B weights are large ~60+ GB on disk ; the 4 TB NVMe is fine, but watch free space if you keep multiple quants | | Access | A Hugging Face account + access token for openai/gpt-oss-120b | Set your token once: export HF TOKEN="hf xxxxxxxxxxxxxxxxx" 3. Pick an inference engine Three viable paths, from easiest to highest-throughput. All three serve an HTTP API you can point a client at. | Engine | Effort | API exposed | Best for | |---|---|---|---| Ollama | Lowest | OpenAI-compatible | Quick start, single user | llama.cpp | Medium | OpenAI-compatible | Control, tuning, GGUF quants | SGLang | Higher | OpenAI-compatible + Anthropic-compatible via proxy | Best measured throughput on Spark | Community testing on the Spark consistently recommendsllama.cpp or SGLang over Ollamafor throughput on this hardware. Use Ollama to confirm everything works, then move to llama.cpp/SGLang for daily use. 4. Option A — Ollama fastest to first token Pull and run; Ollama fetches the official MXFP4 build ollama pull gpt-oss:120b ollama run gpt-oss:120b Ollama exposes an OpenAI-compatible endpoint at http://localhost:11434/v1 . Caveats: - Ollama defaults to a 4096-token context . Raise it for real coding work see model/Modelfile context settings . - Performance is acceptable for testing but typically below a tuned llama.cpp/SGLang setup. 5. Option B — llama.cpp recommended for control Build llama.cpp with CUDA support for the Blackwell GPU, then serve a GGUF build of the model. ~/llama.cpp/build/bin/llama-server \ -m ~/.cache/llama.cpp/gpt-oss-120b/gpt-oss-120b.gguf \ -c 16384 \ context length — tune to your workload see notes -ngl 999 \ offload all layers to the Blackwell GPU --flash-attn on \ enable flash attention --no-mmap \ see mmap note below --kv-unified \ single shared KV buffer --jinja \ use the model's chat template -ub 2048 \ micro-batch size for prompt processing --host 0.0.0.0 \ --port 8005 Flag rationale: -ngl 999 — force all layers onto the GPU. On unified memory this keeps everything in the fast path. --no-mmap — there is a known mmap issue on the Spark that inflates model load time reported ~5× . Disabling mmap fixes load times. --flash-attn on — standard attention speedup for transformer inference. -c context — directly trades off against memory and speed. Larger context grows the KV cache and reduces tok/s. On a comparable small-active MoE, throughput dropped from ~20–25 tok/s at 16K context to ~15–17 tok/s at 32K. Start at 16K and only raise it if your task needs it. -ub 2048 — larger micro-batch improves prompt-processing prefill throughput. Endpoint: http://