# Owning Inference - Qwen3.6 on DGX Spark for real coding > Source: > Published: 2026-07-07 02:01:07+00:00 # Owning Inference - Qwen3.6 on DGX Spark for real coding ### How I got Qwen3.6-27B running locally on a DGX Spark with tools, reasoning, and MTP - and used it to ship real code. If you spend as much time on Hacker News as I do, you’ve surely noticed an uptick in conversations around local models, skyrocketing inference costs, shifting focus on Qwen (DeepSeek, Mistral and the likes) for coding tasks and a general frustration around navigating the AI-coding-at-scale-without-going-broke problem. On the other hand, running local models is no small feat by itself. I’m no AI researcher and my (rather naive) assumption when I got my DGX Spark a few months back was that local inference setup would be akin to pulling a Docker image, typing `ollama serve` and pointing my OpenCode to the endpoint. Once done, I’d have infinite tokens at my disposal and have agents completing work 24/7. Needless to say, I opened the Pandora’s box there and now I’m deep into the rabbit hole of figuring out what a functional local LLM stack looks like. A recent [1,190-point HN thread on Qwen 3.6 27B](https://news.ycombinator.com/item?id=48721903) provides a deep insight into the state of local models as of writing this post. Yes, we can now run dense models on a local setup but the true value remains to be seen. That’s exactly what I’ve been trying to do. My goal is to get real, continuous and high quality work done by local models. This post covers the progress I’ve made so far on this journey. My previous post in this series ([Two Qwen3 Models on One DGX Spark](https://devashish.me/p/two-qwen3-models-on-one-dgx-spark)) was about setting up a two model stack on the DGX. This post answers the follow-up: does this stack write code that ships? At present I have `Qwen/Qwen3.6-27B-FP8` as the sole active model on my DGX Spark, serving reasoning + tools + MTP at 256K context, and shipping work into my open-source (OSS) project - [Clawrium](https://github.com/ric03uec/clawrium). This post is broken down into two sections. The first section talks about the journey of getting the stack working and second one shows how this stack is put to work on a real project. *🛠️ I’ve documented the blog writup methodology in the footnotes*.[1](#footnote-1) ## The final stack (for the impatient) **Host:** DGX Spark GB10 (Grace Blackwell ARM64 (aarch64),`sm_121` compute capability), ~119 GiB unified memory**Model:**`Qwen/Qwen3.6-27B-FP8` at 262,144-token context**Image:**`nvcr.io/nvidia/vllm:26.06-py3` with one pip pin (see Wall 3)**Proxy:** LiteLLM v1.88.1 at`:4000` with`drop_params: true` **Runner:** Model Runner**V1**(V2 breaks — see Wall 2)** Flags:**`--enable-prefix-caching --enforce-eager --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 --speculative-config '{"method":"mtp","num_speculative_tokens":1}'` **Clients:** pi.dev (editor) and Hermes agents Now the walls, in the order I hit them to get this working. ## Why Change Models? Before that, a breif overview of the problems I was facing. My [previous setup ](https://www.devashish.me/p/two-qwen3-models-on-one-dgx-spark)on the same hardware was running two models Qwen3-Next-80B-Instruct-FP8 Qwen3-4B-Instruct And before I moved to the current model, I also tried [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B). Pretty much same problem with each of these 4B-Instruct was anyway not suited for large tasks but I tried solving simple bugs with the other two models. I hoped to get better quality with 35B but since it was not a reasoning model and with only 3B active parameters, it failed on these tasks. I hooked these up with Hermes and OpenCode and tried with different prompt sizes, system prompts and other tweaks but they didn’t do much beyond basic summarization tasks. The Hacker News post I mentioned earlier triggered my curiosity to use a dense model for the same task with the same rig. ## Wall 1 - NVIDIA GPU Cloud (NGC) image too old for `qwen3_5` Dropped the model YAML onto the existing `nvcr.io/nvidia/vllm:25.11-py3` image that had been serving Qwen3-Next-80B fine for weeks. The container crashed on start: ``` pydantic_core._pydantic_core.ValidationError: 1 validation error for ModelConfig Value error, The checkpoint you are trying to load has model type `qwen3_5` but Transformers does not recognize this architecture. ``` Qwen3.6 is `Qwen3_5ForConditionalGeneration` — a dense 27B with hybrid attention plus Mamba/GDN layers. The `25.11-py3` image ships a Transformers old enough to predate `qwen3_5` . Upgrading Transformers alone breaks the CUDA (NVIDIA’s Compute Unified Device Architecture) + vLLM + Transformers version matrix that NGC pins together. Bumped the image to `nvcr.io/nvidia/vllm:26.06-py3` (vLLM `0.22.1` , Transformers `5.6.0` , arm64 manifest, FP8 kernels for `sm_121` ). The pull took 35 minutes and looked stuck — 20 GiB across ~9 concurrent CloudFront connections is just the actual wall time. **Lesson**: NGC image tags bundle a version matrix. Don’t upgrade individual components inside a running container. Bump the whole tag. ## Wall 2 - the red herring that hid the real crash New image, `qwen3_5` recognized, weights loaded, torch.compile ran, engine died: ``` ERROR EngineCore failed to start. File "torch/_ops.py", line 1408, in _get_packet op, overload_names = torch._C._jit_get_operation(qualname) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 181 ``` Every guide on the internet ties this UnicodeDecodeError to torch.compile on aarch64. I ran with that hypothesis and added `--enforce-eager` to skip torch.compile entirely. Same crash. `--enforce-eager` was working — the log confirmed `Enforce eager set, disabling torch.compile and CUDAGraphs` — but the crash was unaffected. Read the traceback more carefully. The crash was inside `torch.library._del_library` — that’s **Python interpreter shutdown code**, not init. Something else killed the engine, then the shutdown-cleanup path fired the visible error. Tightened the grep and found the real primary error earlier in the log: ``` AssertionError: Model Runner V2 has not yet supported mamba_cache_mode='align'. ``` Chained together: Qwen3.6-27B is a hybrid attention + Mamba model. `--enable-prefix-caching` forces`mamba_cache_mode='align'` on hybrid models — the only cache layout the linear-attention layers support with prefix caching.Model Runner V2 in vLLM 0.22 is a rewrite that hasn’t reimplemented align-mode prefix caching for hybrids ( [vllm#26201](https://github.com/vllm-project/vllm/issues/26201),[vllm#38041](https://github.com/vllm-project/vllm/issues/38041)).The NGC (NVIDIA GPU Cloud, the `nvcr.io` container registry) 26.06-py3 compose template ships`VLLM_USE_V2_MODEL_RUNNER: "1"` as a default. That flips the switch that trips the assertion.The engine crashes during `_initialize_kv_caches` , the interpreter shuts down,`torch.library._del_library` iterates a garbled JIT (just-in-time) op registry — UnicodeDecodeError, fully downstream. Fix: delete one env line from the compose template. V1 is the documented fallback and matches the vLLM Recipes reference command. **Lesson**: when Python shuts down mid-crash, the visible error is a shutdown artifact. `grep -B2 -A20 "ERROR .* failed to start"` before you form any hypothesis. I lost about half a day to the wrong one. ## Wall 3 - HTTP 500 (Internal Server Error) on every request (Prometheus + FastAPI 0.137) Engine came up clean. But every chat completion returned 500: ``` {"error":{"message":"'_IncludedRouter' object has no attribute 'path'", "type":"InternalServerError","code":500}} ``` FastAPI 0.137 (May 2026) restructured `include_router()` to wrap routers in `_IncludedRouter` objects that don’t expose `.path` . `prometheus-fastapi-instrumentator <= 8.0.0` unconditionally reads `.path` in its middleware. Every request through the middleware raises `AttributeError` , which bubbles to a 500. NGC 26.06-py3 bundles `fastapi >= 0.137` with `prometheus-fastapi-instrumentator == 8.0.0` — the exact broken pair. Upstream fixes exist ([instrumentator 8.0.1](https://github.com/trallnag/prometheus-fastapi-instrumentator/releases), vLLM pull requests (PRs) [#45594](https://github.com/vllm-project/vllm/pull/45594) / [#45629](https://github.com/vllm-project/vllm/pull/45629)). None of them are in the `26.06-py3` build. No newer NGC tag exists. No vLLM CLI flag disables the FastAPI middleware. Fix: a two-line derived Dockerfile that pins the instrumentator forward: ``` FROM nvcr.io/nvidia/vllm:26.06-py3 RUN pip install --no-cache-dir "prometheus-fastapi-instrumentator>=8.0.1" ``` Built locally, tagged `vllm-inx:26.06-py3-patched` . Model YAML `image:` swapped. Chat completion returned 200. **Lesson**: NGC bundles a whole ecosystem tightly. When FastAPI does a routing refactor, middlewares break inside the container even if they’ve shipped a fix. A thin derived Dockerfile with a single pip pin is cheap enough to be the standard workaround. ## Wall 4 - `hermes` parser doesn’t recognize Qwen3.6’s tool emission Baseline serve closed. In the previous post ([Two Qwen3 Models on One DGX Spark](https://www.devashish.me/p/two-qwen3-models-on-one-dgx-spark)), `hermes` worked with Qwen3-Next-80B-Instruct; carrying that forward here, I added tool calling with the same parser: ``` - --enable-auto-tool-choice - --tool-call-parser - hermes ``` Ran a `calculator` tool test. `tool_calls` was `null` . The call text was in `.content` : ``` "content": "...\n\n\n\n42 * 17\n\n\n", "tool_calls": null ``` The model was emitting a tool call — in **Qwen3-native XML** (`` ), not **Hermes JSON** (`{"name":"calculator","arguments":{...}}` ). The parser doesn’t match, everything falls through as raw text, the harness silently thinks the model refused the tool. NGC 26.06-py3 ships two Qwen3 parsers registered as `qwen3_xml` and `qwen3_coder` . Qwen3.6 emits the XML form: ``` - - hermes + - qwen3_xml ``` Re-deploy. Structured tool call recognized, `finish_reason: tool_calls` , `content` mostly clean (reasoning still leaked — the next phase’s target). **Lesson**: tool parsers map to emission formats, not model families. Qwen 2.5 Coder, Qwen3-Coder, and Qwen3.6 all pick different parsers. Run a `calculator` -tool test on raw output before you wire any agent harness. The failure is silent. ## Reasoning + tools together Added `--reasoning-parser qwen3` to route `...` into `.reasoning_content` . The critical simultaneous-use test asks the model to think, then call a tool: ``` "reasoning_content": "The user wants to know 42 times 17. I need to use the calculator tool...", "content": null, "tool_calls": [ {"function": {"arguments": "{\"expression\": \"42 * 17\"}", "name": "calculator"}, ...} ], "finish_reason": "tool_calls" ``` Reasoning populated, `content: null` (no leakage), `tool_calls[0]` populated, `finish_reason: tool_calls` . That’s the shape a coding-agent harness wants. ## MTP - the risk that turned out safe MTP (Multi-Token Prediction speculative decoding) was the highest-risk phase because a draft head that doesn’t respect structured-output boundaries corrupts either the tool XML or the `` tags. Startup log after adding the flag: ``` Resolved architecture: Qwen3_5MTP Loading drafter model... Detected MTP model. Sharing target model {embedding,lm_head} weights with the draft model. ``` Architecture flipped from `Qwen3_5ForConditionalGeneration` to `Qwen3_5MTP` — the FP8 checkpoint ships the draft head embedded. Re-ran the reasoning + tool acceptance test. Same clean shape. MTP did not corrupt tool tags or `` boundaries. **Lesson**: read the resolved architecture line in the startup log. `Qwen3_5MTP` vs `Qwen3_5ForConditionalGeneration` tells you whether MTP is actually active. The flag can be accepted without the checkpoint having the head — silent no-op. ## Wall 5 - pi sends `reasoning_effort` , LiteLLM 400s Wired the pi.dev agent harness with a new provider extension pointed at `http://:4000/v1` , model `Qwen3.6-27B` , `reasoning: true` . First call: ``` 400 litellm.UnsupportedParamsError: openai does not support parameters: ['reasoning_effort'], for model=Qwen/Qwen3.6-27B-FP8. To drop these, set `litellm.drop_params=True` ``` Pi’s `reasoning: true` flag emits an OpenAI-style `reasoning_effort` param with every request. The LiteLLM route maps to a generic `openai/*` upstream, and OpenAI’s SDK strictly refuses `reasoning_effort` for non-o1 models. LiteLLM propagates the strict validation as a 400. This is orthogonal to vLLM’s own reasoning behavior — `--reasoning-parser qwen3` operates on the model’s `` output tokens server-side; `reasoning_effort` is a client-facing param LiteLLM doesn’t route by default. One-line fix in the LiteLLM config: ``` litellm_settings: drop_params: true ``` LiteLLM now silently strips unsupported params before forwarding. **Lesson**: `drop_params: true` is load-bearing for any LiteLLM deployment that serves multiple client SDKs. pi, aider, opencode, Continue, and Cursor each pass different reasoning-related params (`reasoning_effort` , `thinking` , `reasoning` , `thinking_budget` ). Without `drop_params` , every new client is a fresh 400 to debug. Also worth a mention: **GB10’s unified memory pool doesn’t report the way discrete GPUs do.** `nvidia-smi --query-gpu=memory.used` returns `[N/A]` . Use `nvidia-smi --query-compute-apps=used_memory --format=csv` and sum. Same intent, different query. Update any monitoring or pre-flight gates written against discrete-VRAM semantics. ### The generalized rules Five lessons that generalize past this box. **The visible error is not the primary error when Python shuts down.** Grep for`EngineCore failed to start` first; the`_del_library` UnicodeDecodeError is downstream noise.**NGC image tags bundle a version matrix.** Don’t upgrade individual components. Bump the whole tag or write a thin derived Dockerfile.**Model Runner V2 doesn’t support hybrid-attention align-mode prefix caching yet.** If you’re serving Qwen3.5/3.6, MiniMax M2, or any Mamba/GDN model, unset`VLLM_USE_V2_MODEL_RUNNER` .**Tool parsers map to emission formats, not model families.** Run a`calculator` -tool test on raw output before wiring any agent harness. The failure is silent — the call lands in`.content` .`drop_params: true` **in LiteLLM is load-bearing.** Client SDKs pass different reasoning params; the OpenAI upstream rejects unknown ones.`drop_params` decouples clients from upstream opinion. ## Putting it all together This work only matters if the stack survives contact with a real repo. [Clawrium](https://github.com/ric03uec/clawrium) is my open-source agent fleet manager where this stack is routing to. I’m using this project to benchmark the stack with most of the new changes. This is a non-trival project with real users so the objective is to dogfood my stack against something I have an incentive to keep stable. I started with small PRs and some documentation tasks. I’ll give it a few weeks to settle down before graduating to more long running tasks. Addtionally, I’ve also onboarded a Hermes agent to use the same stack. This agent was previously using GLM 5.1 on Openrouter. ### Evidence: code that shipped into Clawrium Here are concrete Clawrium pull requests that went through the local-Qwen path: [#859 — fix(gui): return 502 for lifecycle failures, surface errors in frontend (#712)](https://github.com/ric03uec/clawrium/pull/859)[#855 — fix(#753): validate agent_name at sync boundary before path interpolation](https://github.com/ric03uec/clawrium/pull/855) One important nuance: not every artifact is an execution artifact. Some are planning artifacts that support the same workflow: [#858 — docs(#712): implementation plan for GUI lifecycle 502 fix (plan-only)](https://github.com/ric03uec/clawrium/pull/858)Hermes agent using this setup self-published a release notes Blog: [https://ric03uec.github.io/clawrium/blog/v26.7.0-release](https://ric03uec.github.io/clawrium/blog/v26.7.0-release) ## The workflow As much as this model seems promising, the rule of thumb is: Don’t ask it for any non-trivial planning. Ask it to write the change once the plan is concrete. This also requires a change in approach. Granularity and precision of the plan defines the success of a local model. **Plan with Codex/Claude/GLM.** Feed the issue, the file tree, and the acceptance criteria. Output: an ordered task list with file paths, function signatures, and test expectations.**Execute with local Qwen.** Feed the task list to pi.dev or OpenCode pointed at the LiteLLM proxy. The model reads the codebase (tools), applies diffs (tools), runs tests (tools). All at 256K context with MTP speculative decoding.**Review locally, then ship.** PR draft, review pass, merge. The frontier is called for planning, not for typing.**Harness choice matters**. Opencode remains my daily driver but as much as it makes model switching easier, it comes with real bloat. Going for local LLMs means obsessing over keeping your stack lean. Starting with the harness. I’m getting good results with[pi.dev.](https://pi.dev)**Code without validation is useless.** I haven’t ported my validation harness yet to use the local model (yet). For true cost efficiencies*both*generation and validation need to be localised. That’s the actual split I’m aiming for: frontier models plan, local Qwen executes, and the repo accumulates both kinds of output. ### Trends so far The stack usage is around 50M tokens so far over the past weekend. I’ll start adding more tasks to the queue this week and report updated numbers soon. Qwen3.6-27B-FP8 on the DGX Spark sustained about **14-16 tok/s decode throughput on active two-request workloads**. - Prompt ingest showed bursty but high prefill throughput, with aggregate effective prefill around 12.6k tok/s. - Prefix caching was highly effective at roughly 95% hit rate. - Average TTFT was about 8.5s, and average end-to-end latency was about 42s, driven mostly by long decode phases on very large prompts. - The observed average prompt size was huge at about 78.8k tokens/request. ## Closing Local inference is finally getting interesting because it can survive contact with real work. It has to reason, call tools, survive a real repo, and ship code I’d actually merge. That’s what this DGX Spark stack is starting to do. Frontier models still do the heavy planning. But once the task is concrete, local Qwen is getting close to being the machine I trust to execute it. That’s the future I want: owned inference, owned context, owned workflow. And this is the first setup I’ve had that feels like it might actually get there. Subscribe if you’re wiring local models into a real code path. I’ll continue documenting my journey and would love to hear yours. [1](#footnote-anchor-1) #### Blog Methodology **Research**: 2-week rollout on the DGX Spark, culminating in a 690-line internal retrospective and a merged infra PR (872 additions, 11 files). The Clawrium codebase is now routed through the stack. Detailed notes at each step kept in a JOURNEY.md file with timestamps, my prompts and results.**Drafting process**: Fed Claude the HN thread, the rollout plan, my raw JOURNEY logs, and the Clawrium PR list. Asked for a lessons-learned draft that walked each wall in the order I hit it.**Author rewrite**: ~% rewritten. Every error message, flag, version number, and vLLM issue reference is verified against the raw log lines from the rollout.**Fully mine**: Narrative arc, takeways and commentary. The “Codex plans, local executes” split, the calculator-tool-test heuristic, the trap selection, all trade-off analysis and examples.**Fully LLM**: transitions between walls, the final numbered-lessons condensation.