{"slug": "show-hn-qwen-reasoning-traces-trainer-and-inference-server", "title": "Show HN: Qwen Reasoning Traces Trainer and Inference Server", "summary": "A developer released a pure-Go, cgo-free toolkit for running and fine-tuning the Qwen3.8 27B model in its Q4_K_M GGUF quantization, including a full GGUF/quant stack, a CPU and Vulkan compute backend, the Qwen3.8 forward pass with full attention and GatedDeltaNet, abliteration, and a QLoRA SFT dataset pipeline. The package ships a `qwen-trainer` CLI with `inspect`, `run`, `abliterate`, `search`, `evaluate`, `dataset` and `train` commands, plus a `qwen-server` binary exposing an OpenAI-compatible API with `GET /v1/models`, `POST /v1/chat/completions` and `GET /health`. The server serializes complete forward passes through a Vulkan execution guard so requests interleave between decode steps without corrupting shared command and descriptor state, and GPU kernels from different requests do not execute in parallel.", "body_md": "A pure-Go, cgo-free toolkit for the **Qwen3.8 27B** model in its **Q4_K_M** GGUF\nquantization. It implements a complete GGUF/quant stack, a CPU + Vulkan compute\nbackend, the Qwen3.8 forward pass (full attention + GatedDeltaNet), abliteration,\nand a dataset pipeline for SFT Training (QLoRA).\n\n- Go 1.27 (build and tests).\n- `glslc` and`spirv-val` to rebuild the committed SPIR-V shaders.\n- A Vulkan device for GPU acceleration and the GPU tests (the CLI selects it automatically when it has enough memory).\n- `python3` with venv support only for the external`verify-*` tests.\n\n1. Download **Qwen3.8-27B** (Unsloth Dynamic**Q4_K_M** ) from Hugging Face:\n\n- Repository: [https://huggingface.co/unsloth/Qwen3.8-27B-GGUF](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF)\n- File: in the **Files and versions** list on the right side of the repo page,\ndownload`Qwen3.8-27B-UD-Q4_K_M.gguf`\n\nPlace it at `models/Qwen3.8-27B-UD-Q4_K_M.gguf`, or let the CLI fetch the pinned\nartifact (with resume + SHA-256 verification) for you:\n\n```\n./build/qwen-trainer download;\n```\n\n1. Export the training data from your Opencode environment and use the session\nhistory as the teacher model. This writes the `manifest.json` ,`sft/<topic>/sessions.jsonl` ,`sft/<topic>/turns.jsonl` ,`subagents/<topic>/...` ,\nand`rl/metadata.jsonl` .\n\n```\nopencode-reasoning-extractor ~/.local/share/opencode /my/own/reasoning-traces;\n```\n\n1. Point the Qwen Trainer at that output dataset directory (see [Usage](#usage) ).\n\n```\n# compile shaders, then build build/qwen-trainer\nmake;\n\n# the command is always the first argument; flags (including --model) follow it\n\n# run the Qwen Trainer\n./build/qwen-trainer inspect --model models/Qwen3.8-27B-UD-Q4_K_M.gguf;\n./build/qwen-trainer run --model models/Qwen3.8-27B-UD-Q4_K_M.gguf --prompt \"The capital of France is\" --generate 8;\n\n# tokenize/validate the extractor output and report loss-mask statistics\n# --dataset is repeatable: pass it once per exported dataset directory\n./build/qwen-trainer dataset --model models/Qwen3.8-27B-UD-Q4_K_M.gguf \\\n  --dataset /path/to/my/own/reasoning-traces/from/opencode;\n```\n\n`--model` is required by every model command (`inspect`, `run`, `abliterate`,\n`search`, `evaluate`, `dataset`, `train`); `download`, `gpu-info`, and `status`\ndo not need it. There is no implicit default model path anymore.\n\nThe compute backend is chosen automatically: Vulkan is used when a device is\npresent and it has enough device memory for the model, otherwise the CPU\nreference runs. `gpu-info` reports the detected device and memory.\n\nSee [Abliteration](#abliteration) for removing refusal directions and for the\nparameter search.\n\n`qwen-server` exposes the same Vulkan/CPU inference pipeline through a minimal\nOpenAI-compatible API. Build it with `make server`:\n\n```\nmake server;\n./build/qwen-server --model models/Qwen3.8-27B-UD-Q4_K_M.gguf --port 8080;\n```\n\nIt serves `GET /v1/models`, `POST /v1/chat/completions`, and `GET /health`.\nUp to `--max-concurrent` requests are admitted at once. Complete forward passes\nare serialized by the Vulkan execution guard, so requests interleave between\ndecode steps without corrupting shared command/descriptor state. GPU kernels\nfrom different requests do not execute in parallel.\n\n```\ncurl -N http://127.0.0.1:8080/v1/chat/completions \\\n  -H 'Content-Type: application/json' \\\n  -d '{\n    \"model\": \"Qwen3.8-27B-UD-Q4_K_M\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"What is 2+2?\"}],\n    \"stream\": true\n  }';\n```\n\nSupported request features include `stream` (SSE), `reasoning_content` deltas,\n`tools`/` tool_calls`, `temperature`, `top_p`, `top_k`, `seed`, `stop`,\n`max_tokens`/` max_completion_tokens`, `reasoning_effort`, and\n`chat_template_kwargs` (`enable_thinking`, `preserve_thinking`). Thinking is\nenabled by default. Tool calls use the Qwen XML format and are returned as\nOpenAI `tool_calls` with JSON arguments.\n\nUseful flags:\n\n| Flag | Values | Default | \n|---|---|---|\n| `--device` | `auto` ,`cpu,` vulkan` | `auto` | \n| `--context-length` | `128k` ,`256k` ,`384k` ,`512k` | (GGUF context) | \n| `--kv-cache-type` | `q4_k` ,`q8_0` | `q4_k` | \n| `--max-concurrent` | (int) | `1` | \n| `--default-max-tokens` | (int) | `1024` | \n| `--speculative` | `true` ,`false` | `true` | \n| `--spec-draft-max` | (int) | `4` | \n| `--spec-draft-min-prob` | (float64) | `0.6` | \n| `--model-id` | (string) | (GGUF filename) | \n| `--api-key` | (string) |  | \n\nEnvironment variables:\n\n| Variable | Description | \n|---|---|\n| `QWEN_SERVER_MODEL` | Model GGUF Path | \n| `QWEN_SERVER_API_KEY` | Bearer API Key | \n\nOn Vulkan the full inference forward runs device-resident, and K/V are stored in the quantized cache and appended with GPU shaders. The prefill chunk size is selected automatically from the context length and device memory.\n\nStreaming usage is included when `stream_options.include_usage` is true.\n\nMTP speculative decoding is enabled by default on both Vulkan and CPU.\n\nContext values above the native GGUF context (256k) select YaRN automatically,\nwith the factor derived from `context / native_context`.\n\nSee [ARCHITECTURE.md](https://github.com/cookiengineer/qwen-reasoning-trainer/blob/master/docs/ARCHITECTURE.md) for the underlying pipeline.\n\nRemove the refusal/safety direction from the weights with Heretic-style directional ablation (projected / magnitude-preserving orthogonal ablation, MPOA), then optionally optimize the ablation hyperparameters.\n\n```\n# 1. Ablate with the default parameters and write a full Q4_K_M GGUF.\n./build/qwen-trainer abliterate --model models/Qwen3.8-27B-UD-Q4_K_M.gguf --out models/abliterated.gguf;\n\n# 2. (optional) search direction index + per-component weight kernels and write\n#    the best trial. Trials persist to the study file and resume automatically.\n./build/qwen-trainer search --model models/Qwen3.8-27B-UD-Q4_K_M.gguf \\\n  --trials 25 \\\n  --sampler tpe \\\n  --study models/search.jsonl \\\n  --out models/search-best.gguf;\n\n# 3. Score the result: refusal-keyword rate and KL divergence from the base.\n./build/qwen-trainer evaluate --model models/abliterated.gguf \\\n  --prompts source/prompts/harmful_behaviors_test.jsonl \\\n  --max-tokens 32;\n```\n\n`source/prompts/` bundles the same datasets Heretic uses, pre-converted to JSON\nLines (one `{\"text\": \"...\"}` object per line). They are compiled into the binary\nwith `go:embed`, so `abliterate` and `search` use them by default with no path\nneeded; override with `--good`/`--bad` (`--eval-good`/`--eval-bad` for `search`).\nFiles may be plain text (one prompt per line) or JSONL (`--prompt-col` selects\nthe column, default `text`).\n\n| File | Rows | Role (default) | \n|---|---|---|\n| `harmless_alpaca_train.jsonl` | 25058 | good prompts / residual directions ( `--good` ) | \n| `harmful_behaviors_train.jsonl` | 416 | bad prompts / residual directions ( `--bad` ) | \n| `harmless_alpaca_test.jsonl` | 6265 | KL divergence evaluation ( `--eval-good` ) | \n| `harmful_behaviors_test.jsonl` | 104 | refusal evaluation ( `--eval-bad` ) | \n\nThe full files are large, so the limits default to Heretic's: `--limit 400`\n(residual prompts per set) and `--eval-limit 100` (evaluation prompts per set).\nPass `--limit 0` / `--eval-limit 0` to use an entire file. Every prompt costs one\nmodel forward, so raising the limits directly increases runtime.\n\nThese mirror `mlabonne/harmless_alpaca` and `mlabonne/harmful_behaviors` as\npublished under `heretic-org` on Hugging Face.\n\n| Flag | Command | Meaning | \n|---|---|---|\n| `--row-norm none\\|pre\\|full` | `abliterate` ,`search` | MPOA row normalization (default `full` ) | \n| `--orthogonalize` | `abliterate` ,`search` | project the direction orthogonal to the good direction (default true) | \n| `--direction-index` | `abliterate` | single global direction layer index ( `-1` = per layer, default) | \n| `--winsorize` | `abliterate` ,`search` | clamp residual components to a quantile in `[0,1)` | \n| `--sampler random\\|halton\\|tpe` | `search` | optimizer (default `random` ) | \n| `--lambda` | `search` | KL weight in the objective `refusal + lambda*KL` (default`1.0` ) | \n| `--seed` ,`--study` | `search` | reproducible seed and resumable JSON Lines checkpoint | \n\nNote: the first-token KL is a weak signal for a thinking model (it emits\n`<think>` first), so small divergence scores are expected on short generations.\n\n`evaluate` scores a model along three independent axes; at least one input is\nrequired.\n\n```\n# Refusal keyword rate (and KL divergence from a base with --base).\n./build/qwen-trainer evaluate --model models/abliterated.gguf \\\n  --prompts source/prompts/harmful_behaviors_test.jsonl --max-tokens 32 \\\n  --base models/Qwen3.8-27B-UD-Q4_K_M.gguf;\n\n# Corpus perplexity over a text file, scored in windows of N tokens.\n./build/qwen-trainer evaluate --model models/Qwen3.8-27B-UD-Q4_K_M.gguf --perplexity corpus.txt --window 512;\n\n# Multiple-choice accuracy over JSONL {context, choices, answer} records.\n./build/qwen-trainer evaluate --model models/Qwen3.8-27B-UD-Q4_K_M.gguf --tasks tasks.jsonl --tasks-verbose;\n```\n\nPerplexity uses non-overlapping windows (default 512 tokens, `--window 0` for\nthe whole input); the window bounds peak memory at `vocab * window * 4` bytes,\nso it is the quality/memory knob. Because it computes full LM-head logits for\nevery window it is considerably slower per token than `run`. Task accuracy is\nscored by the length-normalized log-likelihood of each choice given the context\n(`--tasks-normalize=false` for raw sums).\n\nThe default suite is pure Go and needs neither a GPU nor Python:\n\n```\nmake test;      # go test ./...\nmake check;     # fmt-check + vet + test (run this before calling anything done)\n```\n\nTo run the suite on a remote GPU host, stream the working tree over SSH and run a make target there:\n\n```\nbash run-tests.sh debby;          # remote make check\nbash run-tests.sh debby test;     # remote make test\n```\n\nTests that need hardware or external data skip automatically, so `make check`\nworks on any machine:\n\n- Vulkan tests skip when no device is present.\n- Real-model tests (tokenizer specials, example building) skip when no GGUF is\npresent under `models/` .\n\nIndependent cross-checks live under [source/e2e-tests/](https://github.com/cookiengineer/qwen-reasoning-trainer/blob/master/source/e2e-tests), one\nisolated `main` program per check. They shell out to reference implementations\nand are never run by `make check`:\n\n```\nmake verify-ref;    # NumPy forward pass vs the Go tiny model (~3e-8 match)\nmake verify-data;   # Jinja2 render of the model chat template + Python `regex`\n                    # evaluation of the qwen35 pre-tokenizer\nmake verify-quants; # our block decoders + encoders vs llama.cpp's ggml-quants.c\n                    # (needs a C compiler)\n```\n\nEach is also runnable directly, e.g. `go -C source run ./e2e-tests/verify-ref`.\n\nThe Python references use a project-local virtualenv at `source/e2e-tests/.venv`,\ncreated and populated on demand from each check's own `requirements.txt`\n(`source/e2e-tests/verify-ref/requirements.txt`,\n`source/e2e-tests/verify-data/requirements.txt` — `numpy`, `jinja2`, `regex`).\nThe venv is gitignored.\n\n- `make e2e-venv` creates/populates the venv explicitly.\n- `SYSTEM_PYTHON=python3.x` selects the base interpreter used to build it.\n- `E2E_PYTHON=/path/to/python` runs the references against your own environment.\n\nThe programs can also be driven directly with `QWEN38_REF_PYTHON`:\n\n```\nQWEN38_REF_PYTHON=/path/to/python go -C source run ./e2e-tests/verify-ref\nQWEN38_REF_PYTHON=/path/to/python go -C source run ./e2e-tests/verify-data\ngo -C source run ./e2e-tests/verify-quants     # CC=... to pick a compiler\n```\n\nThe dataset reader is additionally checked against a fixture emitted by the\n`opencode-reasoning-extractor`'s own writer\n(`source/internal/dataset/testdata/extractor-writer/`).\n\nReleased under the MIT License. See [LICENSE.txt](https://github.com/cookiengineer/qwen-reasoning-trainer/blob/master/LICENSE.txt).", "url": "https://wpnews.pro/news/show-hn-qwen-reasoning-traces-trainer-and-inference-server", "canonical_source": "https://github.com/cookiengineer/qwen-reasoning-trainer", "published_at": "2026-11-12 06:19:22+00:00", "updated_at": "2026-11-12 06:55:32.984933+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-infrastructure", "developer-tools"], "entities": ["Qwen3.8 27B", "Unsloth", "Hugging Face", "Vulkan", "OpenAI", "Go 1.27", "QLoRA", "GatedDeltaNet"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-qwen-reasoning-traces-trainer-and-inference-server", "markdown": "https://wpnews.pro/news/show-hn-qwen-reasoning-traces-trainer-and-inference-server.md", "text": "https://wpnews.pro/news/show-hn-qwen-reasoning-traces-trainer-and-inference-server.txt", "jsonld": "https://wpnews.pro/news/show-hn-qwen-reasoning-traces-trainer-and-inference-server.jsonld"}}