{"slug": "two-qwen3-models-on-one-dgx-spark-the-residency-math", "title": "Two Qwen3 models on one DGX Spark: the residency math", "summary": "Alibaba's Qwen3-80B and Qwen3-4B models were successfully co-located on a single NVIDIA DGX Spark using vLLM containers behind a LiteLLM proxy, but the 80B model's inability to emit tool calls in auto mode rendered it incompatible with agent SDKs like Hermes. The setup required careful GPU memory allocation, with gpu_memory_utilization fractions summing below 0.95 to avoid out-of-memory errors.", "body_md": "# Two Qwen3 Models on One DGX Spark: The Residency Math for Local LLM Setup\n\n### The residency math, the gpu_memory_utilization trap, and what to verify first. Notes from my experiments with local LLMs.\n\nMy agent stack with Hermes runs on a workstation. The models run on a DGX Spark on the same LAN. The split is deliberate: the workstation stays responsive, the Spark does the GPU work, and they talk over an HTTP proxy.\n\nSince I started managing the agent fleet through [Clawrium](https://github.com/ric03uec/clawrium), the Hermes count has climbed. More agents on more hosts, more concurrent traffic, all hitting the same Spark. What was a one-laptop, one-model setup is now a small fleet against a single backend — and the shape of the load is exactly what a single-model server can’t serve.\n\nThe Spark served models through ollama for months. It worked. One model up, single config, easy to bring down.\n\nBut ollama owns the card. There’s no per-process memory budget, no `gpu_memory_utilization`\n\nknob, no straightforward way to coresident a heavy model for reasoning and a fast model for quick turns. KV cache management is whatever the underlying llama.cpp backend gives you. PagedAttention isn’t there.\n\nvLLM fixes all of that.\n\nPagedAttention reclaims KV blocks instead of contiguous-pinning them.\n\n`gpu_memory_utilization`\n\ngives you a per-container budget.\n\nOne Spark (GB10, 119.67 GiB unified memory) can run multiple vLLM containers behind a LiteLLM proxy on `:4000`\n\n, and Hermes hits one URL to route to either model. The promise: serve Qwen3-Next-80B-Instruct-FP8 for the heavy work and Qwen3-4B-Instruct-2507 for fast turns, coresident, both reachable from a single endpoint.\n\nThat’s the why. What follows is what it took to make the promise hold.\n\nSpark hardware will happily hold two Qwen3 models if the numbers line up. They didn’t, for several days. That’s where my last weekend went.\n\n### Attempt one: trust the target\n\nFirst 80B config: `gpu_memory_utilization: 0.75`\n\n, `max_model_len: 65536`\n\n, `max_num_seqs: 4`\n\n. vLLM’s KV cache init crashed with *“No available memory for the cache blocks.”* Qwen3-Next is mostly Mamba; the per-block page alignment pushes KV pool demand higher than the ~14 GiB residue after weights.\n\nBumped to 0.85. Now the free-memory check crashed: *“Free memory on device (98.51/119.67 GiB) is less than desired GPU memory utilization (0.85, 101.72 GiB).”* The 4B was already resident at ~16 GiB. The 80B’s 0.85 target was reading the whole card, not what was free.\n\nThat’s the first lesson. `gpu_memory_utilization`\n\ni**s a fraction of total GPU memory, not free memory**.\n\nTwo co-resident vLLM processes need their fractions to sum below ~0.95 to leave room for CUDA framework overhead. If your math assumes free, you’ll oscillate between OOMs and silent KV starvation.\n\nSettled at 0.80 / 32k / 2 for the 80B. Loaded clean. KV pool ~20.8 GiB after weights.\n\n### Attempt two: point Hermes at it\n\nThen Hermes came online and tool calls came back as plain text. `<tool_call>`\n\nJSON sitting inside `content`\n\n. `tool_calls: []`\n\n. `finish_reason: stop`\n\n. Hermes never executed it.\n\nA day of parser triage produced nothing actionable. Both `hermes_tool_parser.py`\n\nand `qwen3xml_tool_parser.py`\n\nlook for `<tool_call>`\n\n(singular). The `<tools>`\n\nplural tag is the system-prompt definition, not the output. The parser wasn’t wrong. The model wasn’t emitting.\n\n`tool_choice: \"required\"`\n\nworked. `tool_choice: \"auto\"`\n\ncame back empty: `tool_calls: []`\n\n, `content: \"\"`\n\n, 619 characters of reasoning inside `<think>`\n\nconcluding *“Alright, that’s it”* without emitting the call.\n\nQwen’s own model card states it plainly: Qwen3-Next-80B-Thinking supports only thinking mode. `enable_thinking: false`\n\nis a structural no-op on this checkpoint. `/no_think`\n\nin the prompt is ignored. The model reasons inside `<think>`\n\n, decides, and never emits.\n\nThat’s an unrecoverable failure for any agent SDK that defaults to `tool_choice: \"auto\"`\n\n. The fix wasn’t a parser flag. It was swapping the whole 80B backbone from Thinking to Instruct.\n\n77 GiB pre-pull. Drain GPU. Bring up with `--enable-auto-tool-choice --tool-call-parser hermes`\n\n, no `--reasoning-parser`\n\n. Three LiteLLM aliases (writer / reviewer / sources) all passed `tool_choice: \"auto\"`\n\ncleanly with `finish_reason: tool_calls`\n\n. Trade accepted: reviewer loses native `<think>`\n\ntraces. Reasoning moved into the prompt.\n\n### Attempt three: the bump that broke coresidency\n\nReviewer agent (running on Hermes) needed 64k context. Bumped the 80B to `0.85 / 65536 / 2`\n\n. 80B loaded healthy. The 4B’s restart loop kicked in 19 times: *“Free memory on device (12.58/119.67 GiB) is less than desired GPU memory utilization (0.12, 14.36 GiB).”*\n\n80B’s actual residency at 0.85 was 101.5 GiB. Plus ~5 GiB CUDA framework overhead. That left ~12.5 GiB free. The 4B needed 14.36 GiB. No room.\n\nToned the 80B back to 0.80, dropped the 4B to `0.10 / 16384 / 8`\n\n. Both came up healthy. The 4B’s `max_model_len`\n\nhad to drop because the 0.10 allocation leaves only ~3.5 GiB for KV pool — 32k single-seq KV demand (~4.8 GiB) doesn’t fit; 16k (~2.4 GiB) does.\n\n### The residency math\n\nThis is the table I wish I’d built on day one:\n\nThree observations from the actuals.\n\nThe 80B’s actual residency at 0.80 ran 8 GiB *under* allocation. That cushion is the only reason the 4B’s restart variability doesn’t break the deployment. At 0.85, the cushion went negative — same hardware, same models, same vLLM build.\n\nThe 4B at 0.10 actually resides at 13.8 GiB, not the 12 GiB the target implies. CUDA framework overhead doesn’t disappear at small allocations.\n\nOn Qwen3-Next specifically, `max_model_len × max_num_seqs`\n\nis dominated by Mamba state alignment, not attention KV. Halving `max_model_len`\n\ndoesn’t halve KV pool demand the way it does on a pure attention model. Plan KV against Mamba page sizes, not against intuition from Llama-class models.\n\nOnce the wiring was complete, LiteLLM showed all the aliases for the same two models running on the spark.\n\n### The insight\n\n`gpu_memory_utilization`\n\nis a snapshot vLLM takes at process start, against total card memory. It is not a target against free memory. CUDA contexts from prior failed attempts can transiently inflate residency and trip the check spuriously. Co-resident processes don’t negotiate — they race.\n\nThe only number that matters is actual residency after both processes have stabilized, measured against the headroom the harder-to-restart model needs to come back from a crash. Target allocations are a planning input; actuals are the ground truth.\n\nFor a two-model Spark deployment, the playbook is: load the bigger model first, let it settle, run `nvidia-smi`\n\nto read actual residency, then size the smaller model’s `gpu_memory_utilization`\n\nagainst the free pool minus ~5 GiB for its own framework overhead. Recheck after both restart cleanly twice.\n\n### The 24-hour action\n\nIf you have a vLLM deployment running right now, pull this:\n\n```\nnvidia-smi --query-gpu=memory.used --format=csv\n```\n\nCompare the actual number to what your `gpu_memory_utilization`\n\ntarget implies. If the two diverge by more than 10%, your sizing model is wrong. Fix it before you ship anything that depends on coresidency — agent stacks, parallel workers, fallback chains. The math has to be empirical, not aspirational.\n\nIf you’re standing up a similar local-LLM stack — DGX Spark(or other hardware), vLLM, multiple coresident models, or wiring a remote agent fleet to a single inference backend — I’d love to compare notes.", "url": "https://wpnews.pro/news/two-qwen3-models-on-one-dgx-spark-the-residency-math", "canonical_source": "https://www.devashish.me/p/two-qwen3-models-on-one-dgx-spark", "published_at": "2026-06-18 16:29:52+00:00", "updated_at": "2026-06-21 13:06:22.727164+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-agents", "ai-tools"], "entities": ["Alibaba", "NVIDIA", "DGX Spark", "Qwen3", "vLLM", "LiteLLM", "Hermes", "Clawrium"], "alternates": {"html": "https://wpnews.pro/news/two-qwen3-models-on-one-dgx-spark-the-residency-math", "markdown": "https://wpnews.pro/news/two-qwen3-models-on-one-dgx-spark-the-residency-math.md", "text": "https://wpnews.pro/news/two-qwen3-models-on-one-dgx-spark-the-residency-math.txt", "jsonld": "https://wpnews.pro/news/two-qwen3-models-on-one-dgx-spark-the-residency-math.jsonld"}}