{"slug": "owning-inference-qwen3-6-on-dgx-spark-for-real-coding", "title": "Owning Inference - Qwen3.6 on DGX Spark for real coding", "summary": "A developer successfully runs the Qwen3.6-27B-FP8 model locally on an Nvidia DGX Spark, achieving reasoning, tool use, and multi-token prediction at 256K context, and uses it to ship real code for an open-source project. The setup overcomes multiple technical hurdles including outdated NGC images, broken model runner versions, and dependency conflicts, demonstrating a functional local LLM stack for continuous coding work.", "body_md": "# Owning Inference - Qwen3.6 on DGX Spark for real coding\n\n### How I got Qwen3.6-27B running locally on a DGX Spark with tools, reasoning, and MTP - and used it to ship real code.\n\nIf you spend as much time on Hacker News as I do, you’ve surely noticed an uptick in conversations around local models, skyrocketing inference costs, shifting focus on Qwen (DeepSeek, Mistral and the likes) for coding tasks and a general frustration around navigating the AI-coding-at-scale-without-going-broke problem.\n\nOn the other hand, running local models is no small feat by itself. I’m no AI researcher and my (rather naive) assumption when I got my DGX Spark a few months back was that local inference setup would be akin to pulling a Docker image, typing `ollama serve`\n\nand pointing my OpenCode to the endpoint. Once done, I’d have infinite tokens at my disposal and have agents completing work 24/7.\n\nNeedless to say, I opened the Pandora’s box there and now I’m deep into the rabbit hole of figuring out what a functional local LLM stack looks like.\n\nA recent [1,190-point HN thread on Qwen 3.6 27B](https://news.ycombinator.com/item?id=48721903) provides a deep insight into the state of local models as of writing this post. Yes, we can now run dense models on a local setup but the true value remains to be seen. That’s exactly what I’ve been trying to do. My goal is to get real, continuous and high quality work done by local models. This post covers the progress I’ve made so far on this journey.\n\nMy previous post in this series ([Two Qwen3 Models on One DGX Spark](https://devashish.me/p/two-qwen3-models-on-one-dgx-spark)) was about setting up a two model stack on the DGX. This post answers the follow-up: does this stack write code that ships? At present I have `Qwen/Qwen3.6-27B-FP8`\n\nas the sole active model on my DGX Spark, serving reasoning + tools + MTP at 256K context, and shipping work into my open-source (OSS) project - [Clawrium](https://github.com/ric03uec/clawrium).\n\nThis post is broken down into two sections. The first section talks about the journey of getting the stack working and second one shows how this stack is put to work on a real project.\n\n*🛠️ I’ve documented the blog writup methodology in the footnotes*.[1](#footnote-1)\n\n## The final stack (for the impatient)\n\n**Host:** DGX Spark GB10 (Grace Blackwell ARM64 (aarch64),`sm_121`\n\ncompute capability), ~119 GiB unified memory**Model:**`Qwen/Qwen3.6-27B-FP8`\n\nat 262,144-token context**Image:**`nvcr.io/nvidia/vllm:26.06-py3`\n\nwith one pip pin (see Wall 3)**Proxy:** LiteLLM v1.88.1 at`:4000`\n\nwith`drop_params: true`\n\n**Runner:** Model Runner**V1**(V2 breaks — see Wall 2)** Flags:**`--enable-prefix-caching --enforce-eager --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":1}'`\n\n**Clients:** pi.dev (editor) and Hermes agents\n\nNow the walls, in the order I hit them to get this working.\n\n## Why Change Models?\n\nBefore that, a breif overview of the problems I was facing. My [previous setup ](https://www.devashish.me/p/two-qwen3-models-on-one-dgx-spark)on the same hardware was running two models\n\nQwen3-Next-80B-Instruct-FP8\n\nQwen3-4B-Instruct\n\nAnd before I moved to the current model, I also tried [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B). Pretty much same problem with each of these 4B-Instruct was anyway not suited for large tasks but I tried solving simple bugs with the other two models. I hoped to get better quality with 35B but since it was not a reasoning model and with only 3B active parameters, it failed on these tasks.\n\nI hooked these up with Hermes and OpenCode and tried with different prompt sizes, system prompts and other tweaks but they didn’t do much beyond basic summarization tasks.\n\nThe Hacker News post I mentioned earlier triggered my curiosity to use a dense model for the same task with the same rig.\n\n## Wall 1 - NVIDIA GPU Cloud (NGC) image too old for `qwen3_5`\n\nDropped the model YAML onto the existing `nvcr.io/nvidia/vllm:25.11-py3`\n\nimage that had been serving Qwen3-Next-80B fine for weeks. The container crashed on start:\n\n```\npydantic_core._pydantic_core.ValidationError: 1 validation error for ModelConfig\n  Value error, The checkpoint you are trying to load has model type `qwen3_5`\n  but Transformers does not recognize this architecture.\n```\n\nQwen3.6 is `Qwen3_5ForConditionalGeneration`\n\n— a dense 27B with hybrid attention plus Mamba/GDN layers. The `25.11-py3`\n\nimage ships a Transformers old enough to predate `qwen3_5`\n\n. Upgrading Transformers alone breaks the CUDA (NVIDIA’s Compute Unified Device Architecture) + vLLM + Transformers version matrix that NGC pins together.\n\nBumped the image to `nvcr.io/nvidia/vllm:26.06-py3`\n\n(vLLM `0.22.1`\n\n, Transformers `5.6.0`\n\n, arm64 manifest, FP8 kernels for `sm_121`\n\n). The pull took 35 minutes and looked stuck — 20 GiB across ~9 concurrent CloudFront connections is just the actual wall time.\n\n**Lesson**: NGC image tags bundle a version matrix. Don’t upgrade individual components inside a running container. Bump the whole tag.\n\n## Wall 2 - the red herring that hid the real crash\n\nNew image, `qwen3_5`\n\nrecognized, weights loaded, torch.compile ran, engine died:\n\n```\nERROR EngineCore failed to start.\nFile \"torch/_ops.py\", line 1408, in _get_packet\n    op, overload_names = torch._C._jit_get_operation(qualname)\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 181\n```\n\nEvery guide on the internet ties this UnicodeDecodeError to torch.compile on aarch64. I ran with that hypothesis and added `--enforce-eager`\n\nto skip torch.compile entirely. Same crash. `--enforce-eager`\n\nwas working — the log confirmed `Enforce eager set, disabling torch.compile and CUDAGraphs`\n\n— but the crash was unaffected.\n\nRead the traceback more carefully. The crash was inside `torch.library._del_library`\n\n— that’s **Python interpreter shutdown code**, not init.\n\nSomething else killed the engine, then the shutdown-cleanup path fired the visible error. Tightened the grep and found the real primary error earlier in the log:\n\n```\nAssertionError: Model Runner V2 has not yet supported mamba_cache_mode='align'.\n```\n\nChained together:\n\nQwen3.6-27B is a hybrid attention + Mamba model.\n\n`--enable-prefix-caching`\n\nforces`mamba_cache_mode='align'`\n\non hybrid models — the only cache layout the linear-attention layers support with prefix caching.Model Runner V2 in vLLM 0.22 is a rewrite that hasn’t reimplemented align-mode prefix caching for hybrids (\n\n[vllm#26201](https://github.com/vllm-project/vllm/issues/26201),[vllm#38041](https://github.com/vllm-project/vllm/issues/38041)).The NGC (NVIDIA GPU Cloud, the\n\n`nvcr.io`\n\ncontainer registry) 26.06-py3 compose template ships`VLLM_USE_V2_MODEL_RUNNER: \"1\"`\n\nas a default. That flips the switch that trips the assertion.The engine crashes during\n\n`_initialize_kv_caches`\n\n, the interpreter shuts down,`torch.library._del_library`\n\niterates a garbled JIT (just-in-time) op registry — UnicodeDecodeError, fully downstream.\n\nFix: delete one env line from the compose template. V1 is the documented fallback and matches the vLLM Recipes reference command.\n\n**Lesson**: when Python shuts down mid-crash, the visible error is a shutdown artifact. `grep -B2 -A20 \"ERROR .* failed to start\"`\n\nbefore you form any hypothesis. I lost about half a day to the wrong one.\n\n## Wall 3 - HTTP 500 (Internal Server Error) on every request (Prometheus + FastAPI 0.137)\n\nEngine came up clean. But every chat completion returned 500:\n\n```\n{\"error\":{\"message\":\"'_IncludedRouter' object has no attribute 'path'\",\n          \"type\":\"InternalServerError\",\"code\":500}}\n```\n\nFastAPI 0.137 (May 2026) restructured `include_router()`\n\nto wrap routers in `_IncludedRouter`\n\nobjects that don’t expose `.path`\n\n. `prometheus-fastapi-instrumentator <= 8.0.0`\n\nunconditionally reads `.path`\n\nin its middleware. Every request through the middleware raises `AttributeError`\n\n, which bubbles to a 500. NGC 26.06-py3 bundles `fastapi >= 0.137`\n\nwith `prometheus-fastapi-instrumentator == 8.0.0`\n\n— the exact broken pair.\n\nUpstream fixes exist ([instrumentator 8.0.1](https://github.com/trallnag/prometheus-fastapi-instrumentator/releases), vLLM pull requests (PRs) [#45594](https://github.com/vllm-project/vllm/pull/45594) / [#45629](https://github.com/vllm-project/vllm/pull/45629)). None of them are in the `26.06-py3`\n\nbuild. No newer NGC tag exists. No vLLM CLI flag disables the FastAPI middleware.\n\nFix: a two-line derived Dockerfile that pins the instrumentator forward:\n\n```\nFROM nvcr.io/nvidia/vllm:26.06-py3\nRUN pip install --no-cache-dir \"prometheus-fastapi-instrumentator>=8.0.1\"\n```\n\nBuilt locally, tagged `vllm-inx:26.06-py3-patched`\n\n. Model YAML `image:`\n\nswapped. Chat completion returned 200.\n\n**Lesson**: NGC bundles a whole ecosystem tightly. When FastAPI does a routing refactor, middlewares break inside the container even if they’ve shipped a fix. A thin derived Dockerfile with a single pip pin is cheap enough to be the standard workaround.\n\n## Wall 4 - `hermes`\n\nparser doesn’t recognize Qwen3.6’s tool emission\n\nBaseline serve closed. In the previous post ([Two Qwen3 Models on One DGX Spark](https://www.devashish.me/p/two-qwen3-models-on-one-dgx-spark)), `hermes`\n\nworked with Qwen3-Next-80B-Instruct; carrying that forward here, I added tool calling with the same parser:\n\n```\n- --enable-auto-tool-choice\n- --tool-call-parser\n- hermes\n```\n\nRan a `calculator`\n\ntool test. `tool_calls`\n\nwas `null`\n\n. The call text was in `.content`\n\n:\n\n```\n\"content\": \"...\\n<tool_call>\\n<function=calculator>\\n<parameter=expression>\\n42 * 17\\n</parameter>\\n</function>\\n</tool_call>\",\n\"tool_calls\": null\n```\n\nThe model was emitting a tool call — in **Qwen3-native XML** (`<function=…><parameter=…>`\n\n), not **Hermes JSON** (`<tool_call>{\"name\":\"calculator\",\"arguments\":{...}}</tool_call>`\n\n).\n\nThe parser doesn’t match, everything falls through as raw text, the harness silently thinks the model refused the tool.\n\nNGC 26.06-py3 ships two Qwen3 parsers registered as `qwen3_xml`\n\nand `qwen3_coder`\n\n. Qwen3.6 emits the XML form:\n\n```\n- - hermes\n+ - qwen3_xml\n```\n\nRe-deploy. Structured tool call recognized, `finish_reason: tool_calls`\n\n, `content`\n\nmostly clean (reasoning still leaked — the next phase’s target).\n\n**Lesson**: tool parsers map to emission formats, not model families. Qwen 2.5 Coder, Qwen3-Coder, and Qwen3.6 all pick different parsers. Run a `calculator`\n\n-tool test on raw output before you wire any agent harness. The failure is silent.\n\n## Reasoning + tools together\n\nAdded `--reasoning-parser qwen3`\n\nto route `<think>...</think>`\n\ninto `.reasoning_content`\n\n. The critical simultaneous-use test asks the model to think, then call a tool:\n\n```\n\"reasoning_content\": \"The user wants to know 42 times 17. I need to use the calculator tool...\",\n\"content\": null,\n\"tool_calls\": [\n  {\"function\": {\"arguments\": \"{\\\"expression\\\": \\\"42 * 17\\\"}\", \"name\": \"calculator\"}, ...}\n],\n\"finish_reason\": \"tool_calls\"\n```\n\nReasoning populated, `content: null`\n\n(no leakage), `tool_calls[0]`\n\npopulated, `finish_reason: tool_calls`\n\n. That’s the shape a coding-agent harness wants.\n\n## MTP - the risk that turned out safe\n\nMTP (Multi-Token Prediction speculative decoding) was the highest-risk phase because a draft head that doesn’t respect structured-output boundaries corrupts either the tool XML or the `<think>`\n\ntags. Startup log after adding the flag:\n\n```\nResolved architecture: Qwen3_5MTP\nLoading drafter model...\nDetected MTP model. Sharing target model {embedding,lm_head} weights with the draft model.\n```\n\nArchitecture flipped from `Qwen3_5ForConditionalGeneration`\n\nto `Qwen3_5MTP`\n\n— the FP8 checkpoint ships the draft head embedded. Re-ran the reasoning + tool acceptance test. Same clean shape. MTP did not corrupt tool tags or `<think>`\n\nboundaries.\n\n**Lesson**: read the resolved architecture line in the startup log. `Qwen3_5MTP`\n\nvs `Qwen3_5ForConditionalGeneration`\n\ntells you whether MTP is actually active. The flag can be accepted without the checkpoint having the head — silent no-op.\n\n## Wall 5 - pi sends `reasoning_effort`\n\n, LiteLLM 400s\n\nWired the pi.dev agent harness with a new provider extension pointed at `http://<host>:4000/v1`\n\n, model `Qwen3.6-27B`\n\n, `reasoning: true`\n\n. First call:\n\n```\n400 litellm.UnsupportedParamsError: openai does not support parameters:\n['reasoning_effort'], for model=Qwen/Qwen3.6-27B-FP8.\nTo drop these, set `litellm.drop_params=True`\n```\n\nPi’s `reasoning: true`\n\nflag emits an OpenAI-style `reasoning_effort`\n\nparam with every request. The LiteLLM route maps to a generic `openai/*`\n\nupstream, and OpenAI’s SDK strictly refuses `reasoning_effort`\n\nfor non-o1 models. LiteLLM propagates the strict validation as a 400. This is orthogonal to vLLM’s own reasoning behavior — `--reasoning-parser qwen3`\n\noperates on the model’s `<think>`\n\noutput tokens server-side; `reasoning_effort`\n\nis a client-facing param LiteLLM doesn’t route by default.\n\nOne-line fix in the LiteLLM config:\n\n```\nlitellm_settings:\n  drop_params: true\n```\n\nLiteLLM now silently strips unsupported params before forwarding.\n\n**Lesson**: `drop_params: true`\n\nis load-bearing for any LiteLLM deployment that serves multiple client SDKs. pi, aider, opencode, Continue, and Cursor each pass different reasoning-related params (`reasoning_effort`\n\n, `thinking`\n\n, `reasoning`\n\n, `thinking_budget`\n\n). Without `drop_params`\n\n, every new client is a fresh 400 to debug.\n\nAlso worth a mention: **GB10’s unified memory pool doesn’t report the way discrete GPUs do.** `nvidia-smi --query-gpu=memory.used`\n\nreturns `[N/A]`\n\n. Use `nvidia-smi --query-compute-apps=used_memory --format=csv`\n\nand sum. Same intent, different query. Update any monitoring or pre-flight gates written against discrete-VRAM semantics.\n\n### The generalized rules\n\nFive lessons that generalize past this box.\n\n**The visible error is not the primary error when Python shuts down.** Grep for`EngineCore failed to start`\n\nfirst; the`_del_library`\n\nUnicodeDecodeError is downstream noise.**NGC image tags bundle a version matrix.** Don’t upgrade individual components. Bump the whole tag or write a thin derived Dockerfile.**Model Runner V2 doesn’t support hybrid-attention align-mode prefix caching yet.** If you’re serving Qwen3.5/3.6, MiniMax M2, or any Mamba/GDN model, unset`VLLM_USE_V2_MODEL_RUNNER`\n\n.**Tool parsers map to emission formats, not model families.** Run a`calculator`\n\n-tool test on raw output before wiring any agent harness. The failure is silent — the call lands in`.content`\n\n.`drop_params: true`\n\n**in LiteLLM is load-bearing.** Client SDKs pass different reasoning params; the OpenAI upstream rejects unknown ones.`drop_params`\n\ndecouples clients from upstream opinion.\n\n## Putting it all together\n\nThis work only matters if the stack survives contact with a real repo. [Clawrium](https://github.com/ric03uec/clawrium) is my open-source agent fleet manager where this stack is routing to. I’m using this project to benchmark the stack with most of the new changes. This is a non-trival project with real users so the objective is to dogfood my stack against something I have an incentive to keep stable. I started with small PRs and some documentation tasks. I’ll give it a few weeks to settle down before graduating to more long running tasks.\n\nAddtionally, I’ve also onboarded a Hermes agent to use the same stack.\n\nThis agent was previously using GLM 5.1 on Openrouter.\n\n### Evidence: code that shipped into Clawrium\n\nHere are concrete Clawrium pull requests that went through the local-Qwen path:\n\n[#859 — fix(gui): return 502 for lifecycle failures, surface errors in frontend (#712)](https://github.com/ric03uec/clawrium/pull/859)[#855 — fix(#753): validate agent_name at sync boundary before path interpolation](https://github.com/ric03uec/clawrium/pull/855)\n\nOne important nuance: not every artifact is an execution artifact. Some are planning artifacts that support the same workflow:\n\n[#858 — docs(#712): implementation plan for GUI lifecycle 502 fix (plan-only)](https://github.com/ric03uec/clawrium/pull/858)Hermes agent using this setup self-published a release notes Blog:\n\n[https://ric03uec.github.io/clawrium/blog/v26.7.0-release](https://ric03uec.github.io/clawrium/blog/v26.7.0-release)\n\n## The workflow\n\nAs much as this model seems promising, the rule of thumb is: Don’t ask it for any non-trivial planning. Ask it to write the change once the plan is concrete. This also requires a change in approach. Granularity and precision of the plan defines the success of a local model.\n\n**Plan with Codex/Claude/GLM.** Feed the issue, the file tree, and the acceptance criteria. Output: an ordered task list with file paths, function signatures, and test expectations.**Execute with local Qwen.** Feed the task list to pi.dev or OpenCode pointed at the LiteLLM proxy. The model reads the codebase (tools), applies diffs (tools), runs tests (tools). All at 256K context with MTP speculative decoding.**Review locally, then ship.** PR draft, review pass, merge. The frontier is called for planning, not for typing.**Harness choice matters**. Opencode remains my daily driver but as much as it makes model switching easier, it comes with real bloat. Going for local LLMs means obsessing over keeping your stack lean. Starting with the harness. I’m getting good results with[pi.dev.](https://pi.dev)**Code without validation is useless.** I haven’t ported my validation harness yet to use the local model (yet). For true cost efficiencies*both*generation and validation need to be localised.\n\nThat’s the actual split I’m aiming for: frontier models plan, local Qwen executes, and the repo accumulates both kinds of output.\n\n### Trends so far\n\nThe stack usage is around 50M tokens so far over the past weekend. I’ll start adding more tasks to the queue this week and report updated numbers soon.\n\nQwen3.6-27B-FP8 on the DGX Spark sustained about **14-16 tok/s decode throughput on active two-request workloads**.\n\n- Prompt ingest showed bursty but high prefill throughput, with aggregate effective prefill around 12.6k tok/s.\n\n- Prefix caching was highly effective at roughly 95% hit rate.\n\n- Average TTFT was about 8.5s, and average end-to-end latency was about 42s, driven mostly by long decode phases on very large prompts.\n\n- The observed average prompt size was huge at about 78.8k tokens/request.\n\n## Closing\n\nLocal inference is finally getting interesting because it can survive contact with real work. It has to reason, call tools, survive a real repo, and ship code I’d actually merge.\n\nThat’s what this DGX Spark stack is starting to do. Frontier models still do the heavy planning. But once the task is concrete, local Qwen is getting close to being the machine I trust to execute it. That’s the future I want: owned inference, owned context, owned workflow. And this is the first setup I’ve had that feels like it might actually get there.\n\nSubscribe if you’re wiring local models into a real code path. I’ll continue documenting my journey and would love to hear yours.\n\n[1](#footnote-anchor-1)\n\n#### Blog Methodology\n\n**Research**: 2-week rollout on the DGX Spark, culminating in a 690-line internal retrospective and a merged infra PR (872 additions, 11 files). The Clawrium codebase is now routed through the stack. Detailed notes at each step kept in a JOURNEY.md file with timestamps, my prompts and results.**Drafting process**: Fed Claude the HN thread, the rollout plan, my raw JOURNEY logs, and the Clawrium PR list. Asked for a lessons-learned draft that walked each wall in the order I hit it.**Author rewrite**: ~% rewritten. Every error message, flag, version number, and vLLM issue reference is verified against the raw log lines from the rollout.**Fully mine**: Narrative arc, takeways and commentary. The “Codex plans, local executes” split, the calculator-tool-test heuristic, the trap selection, all trade-off analysis and examples.**Fully LLM**: transitions between walls, the final numbered-lessons condensation.", "url": "https://wpnews.pro/news/owning-inference-qwen3-6-on-dgx-spark-for-real-coding", "canonical_source": "https://www.devashish.me/p/owning-inference-qwen36-on-dgx-spark", "published_at": "2026-07-07 02:01:07+00:00", "updated_at": "2026-07-07 02:42:49.386899+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-infrastructure", "developer-tools", "ai-research"], "entities": ["Qwen", "Nvidia", "DGX Spark", "LiteLLM", "Hermes", "OpenCode", "Clawrium", "Hacker News"], "alternates": {"html": "https://wpnews.pro/news/owning-inference-qwen3-6-on-dgx-spark-for-real-coding", "markdown": "https://wpnews.pro/news/owning-inference-qwen3-6-on-dgx-spark-for-real-coding.md", "text": "https://wpnews.pro/news/owning-inference-qwen3-6-on-dgx-spark-for-real-coding.txt", "jsonld": "https://wpnews.pro/news/owning-inference-qwen3-6-on-dgx-spark-for-real-coding.jsonld"}}