Why Qwen3.6-35B Runs on a NVIDIA DGX Spark and gpt-oss-120B Fought Me Every Step A developer reports that running gpt-oss-120B on an NVIDIA DGX Spark (GB10) failed due to MXFP4 quantization lacking native hardware support on SM121, causing kernel hangs and memory spikes. Switching to Qwen3.6-35B-A3B in FP8, which has well-supported kernel paths, allowed the model to load and serve cleanly. The experience highlights that quantization format compatibility is critical on unified-memory systems like the DGX Spark. Why Qwen3.6-35B Runs on a NVIDIA DGX Spark and gpt-oss-120B Fought Me Every Step A field report from getting a local LLM inference endpoint working on an NVIDIA DGX Spark GB10 / SM121, 128 GB unified memory — including every wall I hit with gpt-oss-120B, why a smaller FP8 model sidestepped all of them, and how to expose the result safely through an nginx reverse proxy on a multihomed server. TL;DR: On a GB10 Spark, the quantization format matters more than raw capability. gpt-oss-120B ships in MXFP4, which has no native hardware support on SM121 and runs through fragile software kernel paths; combined with the Spark's unified memory, that produced a cascade of freezes and crashes. Qwen3.6-35B-A3B in FP8 — smaller, mixture-of-experts, and on a well-supported kernel path — loaded and served cleanly on the first honest attempt. The hardware, and the two traps it sets The DGX Spark is a GB10 Grace Blackwell machine with 128 GB of unified memory shared between CPU and GPU. Two architectural facts shaped everything that followed: Unified memory is shared. vLLM's --gpu-memory-utilization is a fraction of the entire 128 GB pool, not a separate VRAM budget. The default is 0.9 . On a discrete GPU that only touches VRAM; here it starves the host OS. SM121 has no native FP4. Blackwell-class GB10 runs FP4 weights through software decompression kernels Marlin/CUTLASS paths . For MXFP4 models like gpt-oss, those paths are immature and version-sensitive. Neither is obvious until you trip over it. I tripped over both. The gpt-oss-120B saga Wall 1 — the host froze at the default memory setting The first bare vllm serve openai/gpt-oss-120b reserved ~90% of the unified pool ~115 GB , leaving the kernel, Docker, and sshd to fight over the remaining ~13 GB. The box stopped responding to SSH while still answering ping — classic memory starvation, not a crash. The fix is to leave the host real headroom: --gpu-memory-utilization 0.70 ~26 GB free for the OS . On unified memory you never run the 0.9 default. Wall 2 — "it loaded" is not "it serves" With memory tamed, the model loaded and idled happily at ~74 GB used. Then the first inference request wedged the entire host. Loading and serving are different phases with different failure modes, and the first decode is where the GB10-specific kernel problems actually bite. Wall 3 — the MXFP4-on-SM121 problem the real one This is the crux. gpt-oss-120B's weights are MXFP4, and on SM121 vLLM's default backend selection lands on a kernel path that hangs or crashes on first decode. The community has converged on workarounds, but they're entangled with a specific patched build of vLLM + FlashInfer. On the stock NVIDIA NGC container, those workarounds don't all apply, which produced a string of secondary failures: unrecognized arguments: --mxfp4-layers — that flag exists only in the patched build; stock vLLM 0.21.0 rejects it. FLASHINFER ... attention sinks not supported — gpt-oss uses attention sinks, and the stock container's FlashInfer can't do them, so forcing that backend aborted load. The patched build compiles its own FlashInfer that can. Unknown vLLM environment variable: VLLM MXFP4 BACKEND — the marlin-backend env var simply isn't read by this build. Each "fix" from a recipe written against the patched build was a flag the stock container didn't understand. Wall 4 — the memory spike the budget doesn't count Once the flags were stripped back to what stock vLLM accepts, the model loaded and idled at ~74 GB with ~46 GB free — stable. Then the first request did this: 18:55:56 used 76.3G avail 45.4G 18:55:58 used 78.0G avail 43.7G 18:56:00 used 89.8G avail 31.9G 18:56:02 used 110.3G avail 11.4G 18:56:04 used 121.3G avail 0.4G <== host starved A ~47 GB spike on top of the resident model, in six seconds. The cause: CUDA graph capture plus torch.compile firing on the first forward pass — and that memory is not counted against --gpu-memory-utilization . So 0.70 left 46 GB headroom, the spike wanted more, and the host died. The lever that kills it is --enforce-eager , which disables graph capture and compilation at a real throughput cost . That's the trade I'd make to get 120B stable on the stock container — but by this point the smarter move was a different model. A debugging aside: watch available , not free , and watch from elsewhere Two habits saved time. First, the memory metric that matters on Linux is available , not free — during heavy file reads free drops toward zero as the page cache fills, while available which counts reclaimable cache stays healthy. Misreading free as "almost out of memory" sends you chasing ghosts. Second, run your monitoring and test client from a different machine . I was curling the endpoint over SSH on the Spark itself , so when it froze, my client and my shell died with it. A laptop-side memory watcher that streams free /avail and reconnects on drop turns "it froze" into a timestamped, observable event. Why Qwen3.6-35B-A3B-FP8 just worked Switching to Qwen3.6-35B-A3B-FP8 removed every one of those failure classes at once, for three structural reasons: FP8, not MXFP4. FP8 runs on a well-supported kernel path on SM121; vLLM auto-selects a working MoE backend and the model just loads. None of the Marlin/CUTLASS/FlashInfer-sinks drama applies. It fits with room to spare. At ~35 GB of weights against 121 GB, even the CUDA-graph capture spike fits inside the headroom — so there's no first-inference freeze, and you don't even need --enforce-eager . It's a fast MoE. 35B total but only ~3B active parameters per token, so on the bandwidth-bound Spark it decodes quickly for its quality. Benchmarks on Spark report roughly 28–30 tok/s single-stream, scaling to ~150+ tok/s aggregate under concurrency. The lesson generalizes: on GB10, prefer FP8 or a quantization with a mature SM121 kernel over MXFP4, and prefer a model that fits comfortably over one that maxes the unified pool. A 35B FP8 MoE is a far better daily driver here than a 120B MXFP4 model that needs a patched stack and an eager-mode throughput penalty just to stay upright. The working setup .env — secrets, kept out of the compose file Two secrets: the Hugging Face token a read token is enough — you're only downloading and the vLLM API key the bearer token clients must present . Keep them in a .env beside the compose file; Docker Compose auto-loads it for ${VAR} substitution. cd ~/docker/vllm/qwen36 cat .env <<'EOF' HF TOKEN=hf your read token VLLM API KEY=sk-replace-with-a-strong-key EOF chmod 600 .env echo '.env' .gitignore never commit it Generate a strong API key with echo "sk-$ openssl rand -hex 32 " . Fetch the model up front, then share it into the container Don't let the first vllm serve do a multi-gigabyte download as part of startup — stage it once, then mount the cache into the container "download once, mount everywhere" . On a managed Ubuntu/DGX OS box, install the CLI in isolation system Python is externally managed : sudo apt install -y pipx && pipx ensurepath pipx install "huggingface hub cli " pipx inject huggingface hub hf transfer faster large downloads export HF HUB ENABLE HF TRANSFER=1 hf auth login paste the read token hf download Qwen/Qwen3.6-35B-A3B-FP8 lands in ~/.cache/huggingface Run large downloads inside tmux so they survive a dropped SSH session downloads are resumable — re-running hf download continues where it left off . The sharing mechanism is a single volume mount: bind the host cache to the container's cache path. vLLM then finds the weights locally and starts fast, with no network fetch at serve time: volumes: - ~/.cache/huggingface:/root/.cache/huggingface compose.yml services: vllm: image: nvcr.io/nvidia/vllm:26.05.post1-py3 container name: vllm-qwen36 gpus: all network mode: host ipc: host shm size: "16gb" environment: - HF TOKEN=${HF TOKEN} - VLLM API KEY=${VLLM API KEY} bearer token clients must send volumes: - ~/.cache/huggingface:/root/.cache/huggingface command: vllm serve Qwen/Qwen3.6-35B-A3B-FP8 --host 0.0.0.0 --port 8000 --tensor-parallel-size 1 --gpu-memory-utilization 0.70 --max-model-len 32768 --kv-cache-dtype fp8 --max-num-batched-tokens 8192 --enable-prefix-caching --trust-remote-code --enable-auto-tool-choice --tool-call-parser qwen3 coder --reasoning-parser qwen3 restart: "no" flip to unless-stopped once you trust it Notes that matter: No — FP8 is auto-detected from the repo. Passing MXFP4-specific flags here is what broke gpt-oss on stock vLLM. --quantization flag No — the model is small enough that CUDA graphs fit, so you keep full speed. Only add it back if memory climbs on first inference. --enforce-eager , not the VLLM API KEY as an env var --api-key flag, so the secret doesn't show up in ps .- A harmless log warning about "no optimized MoE config for GB10" is expected; it runs fine on auto-tuned defaults. Bring it up and smoke-test it from another machine : HF TOKEN=... VLLM API KEY=... docker compose up -d docker compose logs -f wait for the Uvicorn "listening" line curl http://spark:8000/v1/chat/completions \ -H "Authorization: Bearer $VLLM API KEY" \ -H "Content-Type: application/json" \ -d '{"model":"Qwen/Qwen3.6-35B-A3B-FP8","messages": {"role":"user","content":"12 17"} }' Exposing it: nginx reverse proxy on a multihomed server The clean topology: a multihomed box one foot on the internet, one on the intranet runs nginx, terminates TLS, and forwards inward to the Spark over the LAN. vLLM stays intranet-only and never faces the public internet directly. Start HTTP-only, let certbot add TLS Don't hand-write ssl certificate paths before a cert exists — nginx -t will fail on the missing files. Deploy an HTTP-only server block first, then let certbot edit it in place. /etc/nginx/sites-available/inference.yourdomain.com upstream vllm backend { server 192.168.0.50:8000; spark's INTRANET IP see the .local note keepalive 32; } server { listen 80; listen :: :80; server name inference.yourdomain.com; client max body size 64m; long prompts exceed the 1m default location /v1/ { proxy pass http://vllm backend; proxy http version 1.1; proxy set header Connection ""; proxy set header Host $host; proxy set header X-Real-IP $remote addr; proxy set header X-Forwarded-For $proxy add x forwarded for; proxy set header X-Forwarded-Proto $scheme; client's "Authorization: Bearer