{"slug": "running-gpt-oss-120b-on-a-single-nvidia-dgx-spark-a-practical-guide", "title": "Running GPT-OSS-120B on a Single NVIDIA DGX Spark - A Practical Guide", "summary": "NVIDIA's DGX Spark single-node AI appliance can run OpenAI's open-weight GPT-OSS-120B sparse MoE model at ~50 tokens/s using optimized engines like SGLang or llama.cpp, making it viable as a local coding backend for tools such as Claude Code. The 128 GB unified memory fits the MXFP4 quantized model with room for KV cache, while the low active-parameter count (5.1B per token) avoids bandwidth bottlenecks typical of dense models.", "body_md": "# Running GPT-OSS-120B on a Single NVIDIA DGX Spark - A Practical Guide\n\nNote on the model name:OpenAI’s open-weight family ships as`gpt-oss-20b`\n\nand`gpt-oss-120b`\n\n. There is no`130B`\n\nvariant — this guide targets, which is the one sized to fit the Spark’s unified memory.`gpt-oss-120b`\n\nA practical, single-node setup guide for serving `gpt-oss-120b`\n\nas a local coding backend on the GB10 Grace Blackwell DGX Spark, and wiring it into Claude Code.\n\n## 1. Why this model fits the Spark\n\nThe DGX Spark has **128 GB of coherent unified LPDDR5x** (~119.7 GB addressable by the GPU) but only **~273 GB/s of memory bandwidth**. Token generation is bandwidth-bound, so bandwidth — not capacity — is the limiting factor.\n\n`gpt-oss-120b`\n\nis a good match for two reasons:\n\n**It fits.** In its native**MXFP4** weight format the full model loads into the ~120 GB unified pool with room left for KV cache.**It’s a sparse MoE.** The model has ~117B total parameters but activates only ~5.1B per token. Generation speed scales with*active*parameters against bandwidth, so it runs far faster than a dense model of comparable footprint.\n\nFor reference, on the same box a dense ~32B model is bandwidth-starved (~9–10 tok/s), while small-active MoE models run several times faster. Published `gpt-oss-120b`\n\nresults on the Spark land around **~50 tokens/s** on an optimized engine (SGLang), which is usable for an interactive coding agent.\n\nRule of thumb for the Spark:prefer MoE models with low active-parameter counts; avoid large dense models.\n\n## 2. Prerequisites\n\n| Requirement | Detail |\n|---|---|\n| Hardware | NVIDIA DGX Spark (GB10), 128 GB unified memory |\n| OS | DGX OS (Ubuntu-based, ARM64 / `aarch64` ) |\n| GPU stack | CUDA + drivers preinstalled on DGX OS; Blackwell compute capability `sm_121` |\n| Firmware | Update to a current firmware version before serving (see §6) |\n| Disk | The 120B weights are large (~60+ GB on disk); the 4 TB NVMe is fine, but watch free space if you keep multiple quants |\n| Access | A Hugging Face account + access token for `openai/gpt-oss-120b` |\n\nSet your token once:\n\n```\nexport HF_TOKEN=\"hf_xxxxxxxxxxxxxxxxx\"\n```\n\n## 3. Pick an inference engine\n\nThree viable paths, from easiest to highest-throughput. **All three serve an HTTP API** you can point a client at.\n\n| Engine | Effort | API exposed | Best for |\n|---|---|---|---|\nOllama |\nLowest | OpenAI-compatible | Quick start, single user |\nllama.cpp |\nMedium | OpenAI-compatible | Control, tuning, GGUF quants |\nSGLang |\nHigher | OpenAI-compatible (+ Anthropic-compatible via proxy) | Best measured throughput on Spark |\n\nCommunity testing on the Spark consistently recommendsllama.cpp or SGLang over Ollamafor throughput on this hardware. Use Ollama to confirm everything works, then move to llama.cpp/SGLang for daily use.\n\n## 4. Option A — Ollama (fastest to first token)\n\n```\n# Pull and run; Ollama fetches the official MXFP4 build\nollama pull gpt-oss:120b\nollama run gpt-oss:120b\n```\n\nOllama exposes an OpenAI-compatible endpoint at `http://localhost:11434/v1`\n\n.\n\nCaveats:\n\n- Ollama defaults to a\n**4096-token context**. Raise it for real coding work (see model/Modelfile context settings). - Performance is acceptable for testing but typically below a tuned llama.cpp/SGLang setup.\n\n## 5. Option B — llama.cpp (recommended for control)\n\nBuild llama.cpp with CUDA support for the Blackwell GPU, then serve a GGUF build of the model.\n\n```\n~/llama.cpp/build/bin/llama-server \\\n  -m ~/.cache/llama.cpp/gpt-oss-120b/gpt-oss-120b.gguf \\\n  -c 16384 \\          # context length — tune to your workload (see notes)\n  -ngl 999 \\          # offload all layers to the Blackwell GPU\n  --flash-attn on \\   # enable flash attention\n  --no-mmap \\         # see mmap note below\n  --kv-unified \\      # single shared KV buffer\n  --jinja \\           # use the model's chat template\n  -ub 2048 \\          # micro-batch size for prompt processing\n  --host 0.0.0.0 \\\n  --port 8005\n```\n\n**Flag rationale:**\n\n`-ngl 999`\n\n— force all layers onto the GPU. On unified memory this keeps everything in the fast path.`--no-mmap`\n\n— there is a**known mmap issue on the Spark** that inflates model load time (reported ~5×). Disabling mmap fixes load times.`--flash-attn on`\n\n— standard attention speedup for transformer inference.`-c`\n\n(context) —**directly trades off against memory and speed.** Larger context grows the KV cache and reduces tok/s. On a comparable small-active MoE, throughput dropped from ~20–25 tok/s at 16K context to ~15–17 tok/s at 32K. Start at 16K and only raise it if your task needs it.`-ub 2048`\n\n— larger micro-batch improves prompt-processing (prefill) throughput.\n\nEndpoint: `http://<spark-ip>:8005/v1`\n\n(OpenAI-compatible).\n\n## 6. Option C — SGLang (highest measured throughput)\n\nSGLang has explicit DGX Spark support and produced the best published `gpt-oss-120b`\n\nnumbers (~50 tok/s).\n\nGeneral shape (consult the current SGLang DGX Spark docs for exact flags/container):\n\n```\n# Launch the SGLang server pointing at the 120B weights\npython -m sglang.launch_server \\\n  --model-path openai/gpt-oss-120b \\\n  --host 0.0.0.0 \\\n  --port 30000\n```\n\nNotes:\n\n- The 120B is ~6× the size of the 20B build, so\n**expect longer load times**. - For stability on the larger model,\n**enabling swap memory** on the Spark is recommended. - Endpoint:\n`http://<spark-ip>:30000/v1`\n\n.\n\nFirmware:keep DGX OS current before serving. Via the DGX Dashboard, or on the CLI:\n\n## 7. Verify the server\n\nOpenAI-compatible smoke test against whichever engine you started:\n\n```\ncurl http://localhost:8005/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"gpt-oss-120b\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Write a Python function that returns the nth Fibonacci number.\"}],\n    \"max_tokens\": 256\n  }'\n```\n\nA coherent code response confirms the model is loaded and serving.\n\n## 8. Wire it into Claude Code\n\nClaude Code speaks the **Anthropic /v1/messages API**, while llama.cpp/Ollama/SGLang expose an\n\n**OpenAI-compatible** API. You therefore need one of:\n\n**(a) An Anthropic-compatible endpoint**, exposed directly by the engine or via a bridge,** or****(b) A translation gateway**(e.g.** LiteLLM**) that accepts Anthropic-format requests and forwards them to your OpenAI-compatible server.\n\nClaude Code is pointed at any endpoint with the `ANTHROPIC_BASE_URL`\n\nenvironment variable (this is the official mechanism for routing through a custom endpoint).\n\n### 8a. Direct / bridged endpoint\n\nIf your server (or a thin bridge in front of it) presents an Anthropic-shaped `/v1/messages`\n\nendpoint:\n\n```\nANTHROPIC_BASE_URL=http://localhost:8005 \\\nANTHROPIC_AUTH_TOKEN=dummy \\\nANTHROPIC_DEFAULT_OPUS_MODEL=gpt-oss-120b \\\nANTHROPIC_DEFAULT_SONNET_MODEL=gpt-oss-120b \\\nANTHROPIC_DEFAULT_HAIKU_MODEL=gpt-oss-120b \\\nclaude\n```\n\n`ANTHROPIC_AUTH_TOKEN`\n\ncarries the bearer/gateway token (`dummy`\n\nworks for an open local server that ignores auth).- The\n`ANTHROPIC_DEFAULT_*_MODEL`\n\nvariables map Claude Code’s Opus/Sonnet/Haiku tiers onto your single local model, so every tier resolves to`gpt-oss-120b`\n\n.\n\n### 8b. LiteLLM bridge (for OpenAI-only servers)\n\nRun LiteLLM in front of llama.cpp/Ollama, register the model under `claude-*`\n\naliases, then point Claude Code at LiteLLM’s URL with the same env vars as above. This is the established pattern for using a purely OpenAI-compatible local server with Claude Code on the Spark.\n\n### Persisting and a caching gotcha\n\nAdd the variables to `~/.bashrc`\n\n/`~/.zshrc`\n\n, or to `~/.claude/settings.json`\n\nunder an `env`\n\nblock.\n\n**Prefix-caching note:** Claude Code injects a per-request attribution hash into the system prompt, which can defeat prefix caching and slow throughput. If your serving stack doesn’t handle this automatically, set:\n\n```\n{\n  \"env\": { \"CLAUDE_CODE_ATTRIBUTION_HEADER\": \"0\" }\n}\n```\n\nin `~/.claude/settings.json`\n\n.\n\nLaunch Claude Code and run a small prompt to confirm requests are routing to the Spark.\n\n## 9. Tuning checklist\n\n**Context length is your main lever.** Bigger context = bigger KV cache = lower tok/s and more memory. Right-size it per task (16K is a sane default; raise deliberately).**Stay on MoE.** Don’t swap in dense models on this box expecting similar speed.on llama.cpp to avoid the slow-load bug.`--no-mmap`\n\n**Enable swap** for stability when loading the 120B.**One engine, one quant.** Multiple large GGUF/quant copies fill the NVMe fast.**Watch active-vs-total params**, not total size, when predicting speed.\n\n## 10. Honest expectations vs. “like Opus”\n\nOn a *single* Spark, `gpt-oss-120b`\n\nis the largest coherent, frontier-style reasoning/tool-use model that fits, and it is genuinely usable in a Claude Code loop at ~50 tok/s. It is **not** equivalent to a current frontier closed model. The open models that most directly rival top closed models on agentic coding are trillion-parameter MoEs (e.g. Kimi K2.x, DeepSeek V4-Pro, large GLM MoEs) — those do **not** fit on one Spark and would require clustering two Sparks over the ConnectX-7 200G link or different hardware.\n\nIf you want a *coding-specialized* alternative on the same box, Qwen3-Coder variants (e.g. 30B-A3B, or Qwen3-Coder-Next in FP8/NVFP4) are smaller-active MoEs that run faster and are widely used with Claude Code on the Spark.\n\n### Source anchors\n\n- DGX Spark hardware (GB10, 128 GB unified, 273 GB/s,\n`sm_121`\n\n, DGX OS): NVIDIA / LMSYS / StorageReview reviews. `gpt-oss-120b`\n\non Spark (~50 tok/s, SGLang support, fits 120 GB, swap recommendation): LMSYS DGX Spark + GPT-OSS posts, Ollama Spark performance blog.- llama.cpp flags and the\n`--no-mmap`\n\nload-time bug, context-vs-throughput figures: community Spark engine write-ups. - Dense-vs-MoE throughput contrast and “use llama.cpp / switch to MoE” guidance: NVIDIA developer forum.\n- Claude Code routing (\n`ANTHROPIC_BASE_URL`\n\n,`ANTHROPIC_AUTH_TOKEN`\n\n,`ANTHROPIC_DEFAULT_*_MODEL`\n\n,`CLAUDE_CODE_ATTRIBUTION_HEADER`\n\n): Claude Code authentication docs, vLLM Claude Code integration docs, LiteLLM bridge example.", "url": "https://wpnews.pro/news/running-gpt-oss-120b-on-a-single-nvidia-dgx-spark-a-practical-guide", "canonical_source": "https://corti.com/running-gpt-oss-120b-on-a-single-nvidia-dgx-spark-a-practical-guide/", "published_at": "2026-05-31 11:36:03+00:00", "updated_at": "2026-06-26 12:04:05.795227+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools"], "entities": ["NVIDIA", "DGX Spark", "OpenAI", "GPT-OSS-120B", "SGLang", "llama.cpp", "Ollama", "Claude Code"], "alternates": {"html": "https://wpnews.pro/news/running-gpt-oss-120b-on-a-single-nvidia-dgx-spark-a-practical-guide", "markdown": "https://wpnews.pro/news/running-gpt-oss-120b-on-a-single-nvidia-dgx-spark-a-practical-guide.md", "text": "https://wpnews.pro/news/running-gpt-oss-120b-on-a-single-nvidia-dgx-spark-a-practical-guide.txt", "jsonld": "https://wpnews.pro/news/running-gpt-oss-120b-on-a-single-nvidia-dgx-spark-a-practical-guide.jsonld"}}