XDA ran a piece that stopped me mid-scroll: a 27B open-weights model reverse-engineered a commercial application’s license check — recovered a deliberately obscured crypto key out of ARM64 assembly, caught and corrected its own mistake without being told, and produced a working bypass PoC. In about thirty minutes. On a desktop box. I have (a version of) that box. So I went and set it up.
The model is Qwen3.8-27B. Artificial Analysis has it as the top open-weights model in the 4B–40B class out of 135 models, at 52 on their intelligence index. The hardware in the article was a Lenovo ThinkStation PGX — NVIDIA Grace Blackwell, 128 GB unified memory, 273 GB/s. Mine is an ASUS Ascent GX10, which is the same GB10 silicon in a different box.
I’ve been building toward this for a while. In May I wrote that AI inference costs are the wake-up call for 2026 and 2027. In July I argued that tokens should be NRE, not COGS. Earlier this month I benchmarked the GX10 against a ZBook Ultra G1a on a real agentic coding task and came away convinced the small box is the one you actually reach for.
This post is the missing piece: the actual setup. What follows is the recipe I ran, corrected against a real GX10 bring-up, including the parts that bit me.
Before You Start: Two Things That Will Ruin Your Day #
Disk. You need about 100 GB free. The base container image alone is 57 GB, weights are ~19 GB, caches on top of that.
df -h ~
Free the unified memory first. This is the one that got me. SGLang runs at --mem-fraction-static 0.90
, which claims ~90% of system RAM. Anything else already resident will collide — and on GB10 that collision doesn’t produce a nice error, it produces a hard reboot during CUDA-graph capture.
The usual culprit is ollama
, which pins its model in unified memory and sits there.
nvidia-smi # check the Processes list at the bottom
sudo systemctl stop ollama
sudo systemctl disable ollama # stop it grabbing memory on boot
nvidia-smi # want: "No running processes found"
free -h # want: ~100+ GB free
GB10 quirk:nvidia-smi
reportsMemory-Usage: Not Supported
in the summary table. That’s normal for unified memory — it’s not broken. Read theprocess listat the bottom, or usefree -h
.
Also note free -h
shows about 121 GB, not 128 — firmware and system reserve the rest. At 0.90 that means SGLang takes ~109 GB with comfortable headroom. Don’t “fix” this. 0.95
reboots the box.
OS and Stack Check #
The GX10 ships DGX OS, but a stock Ubuntu 24.04 LTS (arm64) install works fine and is fully supported — driver, CUDA, Docker, Container Toolkit all run natively. Mine’s Ubuntu. Confirm what you have:
cat /etc/os-release # Ubuntu 24.04 (noble) or DGX OS — either is fine
docker version # arm64 engine; 29.x is current
nvidia-ctk --version # NVIDIA Container Toolkit present
nvidia-smi # GB10 visible, driver loaded
A healthy box: NVIDIA GB10
, driver 580.x, CUDA 13.x, Docker 29.x linux/arm64
, Container Toolkit 1.19.x. If Docker or the toolkit are missing:
sudo apt update
sudo apt install -y docker.io
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER" # log out and back in after this
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
Optionally set a Hugging Face token for faster pulls:
echo 'export HF_TOKEN=hf_xxxxxxxxxxxxxxxxx' >> ~/.bashrc && source ~/.bashrc
Clone and Configure #
cd ~
git clone https://github.com/MiaAI-Lab/Qwen3.8-27B-SGLang-DGX-Spark.git
cd Qwen3.8-27B-SGLang-DGX-Spark
cp .env.sample .env
The defaults are sane for code work: NVFP4 weights, native 262K context, YaRN off, 16 concurrent requests.
One hard rule: keep YARN=0 and CONTEXT_LENGTH=262144. DFlash2 is not compatible with YaRN context extension on this build. A larger context leaking into the draft config throws
AttributeError ... max_position_embeddings
at boot and you’ll spend an hour thinking it’s a driver problem. Confirm before you launch:
grep -E '^YARN|^CONTEXT_LENGTH' .env
Also: the repo hardcodes PORT=8888
and SERVED_MODEL_NAME=qwen3.8-27b-sglang
. Overriding either one requires patching start.sh
. I didn’t bother — I use 8888 and read the served model id from /v1/models
. (The patch is a two-line change if you want it; I put it at the end.)
One caveat worth saying out loud: this upstream repo is new and moving. Sanity-check script names and .env
keys against the current README before you run anything. Don’t trust my transcription over their repo.
Build and Launch #
Build the DFlash2 image — a pure local overlay onto the pinned SGLang image, with every overlaid file checksum-verified:
./patch/build-dflash2-image.sh --minimal
First run pulls ~57 GB. Go do something else. Take a nap. Or a walk. It takes a bit, even with a fast network.
Then launch:
DF_EXTRA="--sleep-on-idle" \
IMAGE=lmsysorg/sglang:qwen38-27b-dflash2-minoverlay \
./start-dflash.sh
--sleep-on-idle
matters more than it looks. Without it the SGLang scheduler busy-spins at ~97% CPU doing nothing, which on a small fanless-ish box you can hear and feel.
First boot downloads weights and compiles kernels: 10–20 minutes, mostly silent. Weight load alone is about 5 minutes. Warm restarts are 2–3 minutes. Watch it in another pane:
tail -f .sglang.log
You’re up when you see Uvicorn running on http://0.0.0.0:8888
and Application startup complete
.
Verify #
Fair warning on the first thing that’ll confuse you: curl -s
swallows connection errors, so an empty reply piped to json.tool
gives you Expecting value: line 1 column 1
. That almost always means not listening yet, not crashed. Check docker ps
and the log before you panic.
PORT=8888
MODEL=$(curl -s http://127.0.0.1:$PORT/v1/models \
| python3 -c 'import sys,json;print(json.load(sys.stdin)["data"][0]["id"])')
echo "Serving model id: $MODEL"
curl -s http://127.0.0.1:$PORT/v1/chat/completions \
-H 'Content-Type: application/json' -d "{
\"model\": \"$MODEL\",
\"messages\": [{\"role\":\"user\",\"content\":\"Write an idiomatic Go function that returns the nth Fibonacci number iteratively.\"}],
\"max_tokens\": 300,
\"temperature\": 0.2,
\"chat_template_kwargs\": {\"enable_thinking\": false},
\"stream_options\": {\"include_usage\": true}
}" | python3 -m json.tool
Logs are tail -f .sglang.log
or docker logs -f qwen3.8-27b-sglang
. Stop with ./stop.sh
.
Pointing Your Agents At It #
The server exposes three protocols on http://<gx10-ip>:8888
:
OpenAI Chat Completions—/v1/chat/completions
, works out of the boxOpenAI Responses—/v1/responses
, needs a compat patch for strict SDK clientsAnthropic-style—/v1/messages
For OpenAI-compatible agents — aider, opencode, Cline, Continue, and the ones I’ve been testing lately:
export OPENAI_BASE_URL="http://<gx10-ip>:8888/v1"
export OPENAI_API_KEY="sk-local" # any non-empty string; the server ignores it
aider, concretely:
aider --openai-api-base http://<gx10-ip>:8888/v1 \
--openai-api-key sk-local \
--model openai/qwen3.8-27b-sglang
Claude Code speaks the Anthropic protocol and /v1/messages
is right there:
export ANTHROPIC_BASE_URL="http://<gx10-ip>:8888"
export ANTHROPIC_API_KEY="sk-local"
Treat that one as best-effort. This build 400s on some exotic Responses/message item types — MCP tool-call echoes, web_search_call
echoes, item_reference
. Plain function tools and chat work fully. For predictable agentic coding today, the OpenAI-compatible path is the tested one.
By the way — the XDA reverse-engineering run used the pi
harness, which is worth knowing. The harness matters as much as the endpoint. I’ve made that argument at length.
The Explicit docker run #
If you’d rather own every flag than trust a launcher — and honestly, for something you’re going to run every day, you should read this at least once:
PORT=8888
IMAGE=lmsysorg/sglang:qwen38-27b-dflash2-minoverlay
CACHE=$HOME/Qwen3.8-27B-SGLang-DGX-Spark/.cache
docker run -d --name qwen38-code \
--network host --ipc host --privileged --gpus all --shm-size 32g \
-e HF_HOME=/root/.cache/huggingface \
-e TRITON_CACHE_DIR=/root/.triton \
-v ${CACHE}/huggingface:/root/.cache/huggingface \
-v ${CACHE}/triton:/root/.triton \
${IMAGE} python3 -m sglang.launch_server \
--model-path RadixArk/Qwen3.8-27B-NVFP4 \
--served-model-name qwen38-27b --trust-remote-code \
--tp 1 \
--attention-backend flashinfer --chunked-prefill-size 8192 \
--disable-prefill-cuda-graph --kv-cache-dtype fp8_e4m3 \
--mamba-ssm-dtype bfloat16 --mamba-full-memory-ratio 4.21 \
--mamba-radix-cache-strategy extra_buffer \
--max-mamba-cache-size 64 --max-running-requests 16 \
--context-length 262144 \
--speculative-algorithm DFLASH \
--speculative-draft-model-path z-lab/Qwen3.8-27B-DFlash2 \
--speculative-draft-model-revision 50307d4c4cde6860d4eee73e2547cd786fe8e8a4 \
--speculative-num-draft-tokens 8 \
--reasoning-parser qwen3 --tool-call-parser qwen3_coder \
--sampling-defaults model --enable-metrics --enable-cache-report \
--stream-interval 1 --sleep-on-idle \
--mem-fraction-static 0.90 --host 0.0.0.0 --port ${PORT}
The flags that actually matter for code: --speculative-num-draft-tokens 8
(block size), --tool-call-parser qwen3_coder
(agent function calls — you need this), --stream-interval 1
(per-token SSE), --mem-fraction-static 0.90
(the safe headroom), --kv-cache-dtype fp8_e4m3
(keeps KV memory sane at 262K context).
Setting --served-model-name qwen38-27b
here makes the model id qwen38-27b
regardless of the repo default — which is the easy way around the port/name patch if you’re going explicit anyway.
Conclusion #
Here’s what I keep circling back to.
A 27B model, quantized to 4 bits, running locally, did a static reverse-engineering job on ARM64 assembly — including catching its own error and fixing it — in thirty minutes. Unattended. Off the network.
That is not the story the “local models are toys” crowd has been telling. It’s also not the story the “you need a trillion parameters” crowd has been telling. Both of them are about to be wrong in the same direction, and I’ve said before that today’s models are the worst you’ll ever use.
I’m not claiming this replaces a frontier model for everything. It doesn’t, and I’ll write that post honestly when I’ve run it against enough real work. What I am claiming is that the line between “send this to the cloud” and “run this locally” moved, again, and it moved further than I expected it to this quarter. Every time it moves, the NRE-versus-COGS math gets better for owning the hardware.
Setup is an afternoon, most of it waiting on a 57 GB pull. Go run it and find out where your own line is.
What’s next for me: running this endpoint under the coding agents I’ve been evaluating, and seeing whether a local 27B holds up in a real agentic loop the way it did on a single well-scoped analysis task. Those are very different asks. That’s the post I actually want to write.
Credit where it’s due: the XDA piece by way of Artificial Analysis’s benchmarks is what sent me down this road, and the MiaAI-Lab repo is doing the unglamorous packaging work that makes this a one-afternoon job instead of a one-week job.
If you get this running — or if you hit something my troubleshooting table doesn’t cover — drop me a note on LinkedIn: linkedin.com/in/gherlein. I’ll add it.