Owning Inference - Qwen3.6 on DGX Spark for real coding A developer successfully runs the Qwen3.6-27B-FP8 model locally on an Nvidia DGX Spark, achieving reasoning, tool use, and multi-token prediction at 256K context, and uses it to ship real code for an open-source project. The setup overcomes multiple technical hurdles including outdated NGC images, broken model runner versions, and dependency conflicts, demonstrating a functional local LLM stack for continuous coding work. Owning Inference - Qwen3.6 on DGX Spark for real coding How I got Qwen3.6-27B running locally on a DGX Spark with tools, reasoning, and MTP - and used it to ship real code. If you spend as much time on Hacker News as I do, you’ve surely noticed an uptick in conversations around local models, skyrocketing inference costs, shifting focus on Qwen DeepSeek, Mistral and the likes for coding tasks and a general frustration around navigating the AI-coding-at-scale-without-going-broke problem. On the other hand, running local models is no small feat by itself. I’m no AI researcher and my rather naive assumption when I got my DGX Spark a few months back was that local inference setup would be akin to pulling a Docker image, typing ollama serve and pointing my OpenCode to the endpoint. Once done, I’d have infinite tokens at my disposal and have agents completing work 24/7. Needless to say, I opened the Pandora’s box there and now I’m deep into the rabbit hole of figuring out what a functional local LLM stack looks like. A recent 1,190-point HN thread on Qwen 3.6 27B https://news.ycombinator.com/item?id=48721903 provides a deep insight into the state of local models as of writing this post. Yes, we can now run dense models on a local setup but the true value remains to be seen. That’s exactly what I’ve been trying to do. My goal is to get real, continuous and high quality work done by local models. This post covers the progress I’ve made so far on this journey. My previous post in this series Two Qwen3 Models on One DGX Spark https://devashish.me/p/two-qwen3-models-on-one-dgx-spark was about setting up a two model stack on the DGX. This post answers the follow-up: does this stack write code that ships? At present I have Qwen/Qwen3.6-27B-FP8 as the sole active model on my DGX Spark, serving reasoning + tools + MTP at 256K context, and shipping work into my open-source OSS project - Clawrium https://github.com/ric03uec/clawrium . This post is broken down into two sections. The first section talks about the journey of getting the stack working and second one shows how this stack is put to work on a real project. 🛠️ I’ve documented the blog writup methodology in the footnotes . 1 footnote-1 The final stack for the impatient Host: DGX Spark GB10 Grace Blackwell ARM64 aarch64 , sm 121 compute capability , ~119 GiB unified memory Model: Qwen/Qwen3.6-27B-FP8 at 262,144-token context Image: nvcr.io/nvidia/vllm:26.06-py3 with one pip pin see Wall 3 Proxy: LiteLLM v1.88.1 at :4000 with drop params: true Runner: Model Runner V1 V2 breaks — see Wall 2 Flags: --enable-prefix-caching --enforce-eager --enable-auto-tool-choice --tool-call-parser qwen3 xml --reasoning-parser qwen3 --speculative-config '{"method":"mtp","num speculative tokens":1}' Clients: pi.dev editor and Hermes agents Now the walls, in the order I hit them to get this working. Why Change Models? Before that, a breif overview of the problems I was facing. My previous setup https://www.devashish.me/p/two-qwen3-models-on-one-dgx-spark on the same hardware was running two models Qwen3-Next-80B-Instruct-FP8 Qwen3-4B-Instruct And before I moved to the current model, I also tried Qwen3.6-35B-A3B https://huggingface.co/Qwen/Qwen3.6-35B-A3B . Pretty much same problem with each of these 4B-Instruct was anyway not suited for large tasks but I tried solving simple bugs with the other two models. I hoped to get better quality with 35B but since it was not a reasoning model and with only 3B active parameters, it failed on these tasks. I hooked these up with Hermes and OpenCode and tried with different prompt sizes, system prompts and other tweaks but they didn’t do much beyond basic summarization tasks. The Hacker News post I mentioned earlier triggered my curiosity to use a dense model for the same task with the same rig. Wall 1 - NVIDIA GPU Cloud NGC image too old for qwen3 5 Dropped the model YAML onto the existing nvcr.io/nvidia/vllm:25.11-py3 image that had been serving Qwen3-Next-80B fine for weeks. The container crashed on start: pydantic core. pydantic core.ValidationError: 1 validation error for ModelConfig Value error, The checkpoint you are trying to load has model type qwen3 5 but Transformers does not recognize this architecture. Qwen3.6 is Qwen3 5ForConditionalGeneration — a dense 27B with hybrid attention plus Mamba/GDN layers. The 25.11-py3 image ships a Transformers old enough to predate qwen3 5 . Upgrading Transformers alone breaks the CUDA NVIDIA’s Compute Unified Device Architecture + vLLM + Transformers version matrix that NGC pins together. Bumped the image to nvcr.io/nvidia/vllm:26.06-py3 vLLM 0.22.1 , Transformers 5.6.0 , arm64 manifest, FP8 kernels for sm 121 . The pull took 35 minutes and looked stuck — 20 GiB across ~9 concurrent CloudFront connections is just the actual wall time. Lesson : NGC image tags bundle a version matrix. Don’t upgrade individual components inside a running container. Bump the whole tag. Wall 2 - the red herring that hid the real crash New image, qwen3 5 recognized, weights loaded, torch.compile ran, engine died: ERROR EngineCore failed to start. File "torch/ ops.py", line 1408, in get packet op, overload names = torch. C. jit get operation qualname UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 181 Every guide on the internet ties this UnicodeDecodeError to torch.compile on aarch64. I ran with that hypothesis and added --enforce-eager to skip torch.compile entirely. Same crash. --enforce-eager was working — the log confirmed Enforce eager set, disabling torch.compile and CUDAGraphs — but the crash was unaffected. Read the traceback more carefully. The crash was inside torch.library. del library — that’s Python interpreter shutdown code , not init. Something else killed the engine, then the shutdown-cleanup path fired the visible error. Tightened the grep and found the real primary error earlier in the log: AssertionError: Model Runner V2 has not yet supported mamba cache mode='align'. Chained together: Qwen3.6-27B is a hybrid attention + Mamba model. --enable-prefix-caching forces mamba cache mode='align' on hybrid models — the only cache layout the linear-attention layers support with prefix caching.Model Runner V2 in vLLM 0.22 is a rewrite that hasn’t reimplemented align-mode prefix caching for hybrids vllm 26201 https://github.com/vllm-project/vllm/issues/26201 , vllm 38041 https://github.com/vllm-project/vllm/issues/38041 .The NGC NVIDIA GPU Cloud, the nvcr.io container registry 26.06-py3 compose template ships VLLM USE V2 MODEL RUNNER: "1" as a default. That flips the switch that trips the assertion.The engine crashes during initialize kv caches , the interpreter shuts down, torch.library. del library iterates a garbled JIT just-in-time op registry — UnicodeDecodeError, fully downstream. Fix: delete one env line from the compose template. V1 is the documented fallback and matches the vLLM Recipes reference command. Lesson : when Python shuts down mid-crash, the visible error is a shutdown artifact. grep -B2 -A20 "ERROR . failed to start" before you form any hypothesis. I lost about half a day to the wrong one. Wall 3 - HTTP 500 Internal Server Error on every request Prometheus + FastAPI 0.137 Engine came up clean. But every chat completion returned 500: {"error":{"message":"' IncludedRouter' object has no attribute 'path'", "type":"InternalServerError","code":500}} FastAPI 0.137 May 2026 restructured include router to wrap routers in IncludedRouter objects that don’t expose .path . prometheus-fastapi-instrumentator <= 8.0.0 unconditionally reads .path in its middleware. Every request through the middleware raises AttributeError , which bubbles to a 500. NGC 26.06-py3 bundles fastapi = 0.137 with prometheus-fastapi-instrumentator == 8.0.0 — the exact broken pair. Upstream fixes exist instrumentator 8.0.1 https://github.com/trallnag/prometheus-fastapi-instrumentator/releases , vLLM pull requests PRs 45594 https://github.com/vllm-project/vllm/pull/45594 / 45629 https://github.com/vllm-project/vllm/pull/45629 . None of them are in the 26.06-py3 build. No newer NGC tag exists. No vLLM CLI flag disables the FastAPI middleware. Fix: a two-line derived Dockerfile that pins the instrumentator forward: FROM nvcr.io/nvidia/vllm:26.06-py3 RUN pip install --no-cache-dir "prometheus-fastapi-instrumentator =8.0.1" Built locally, tagged vllm-inx:26.06-py3-patched . Model YAML image: swapped. Chat completion returned 200. Lesson : NGC bundles a whole ecosystem tightly. When FastAPI does a routing refactor, middlewares break inside the container even if they’ve shipped a fix. A thin derived Dockerfile with a single pip pin is cheap enough to be the standard workaround. Wall 4 - hermes parser doesn’t recognize Qwen3.6’s tool emission Baseline serve closed. In the previous post Two Qwen3 Models on One DGX Spark https://www.devashish.me/p/two-qwen3-models-on-one-dgx-spark , hermes worked with Qwen3-Next-80B-Instruct; carrying that forward here, I added tool calling with the same parser: - --enable-auto-tool-choice - --tool-call-parser - hermes Ran a calculator tool test. tool calls was null . The call text was in .content : "content": "...\n