Show HN: Qwen Reasoning Traces Trainer and Inference Server

A developer released a pure-Go, cgo-free toolkit for running and fine-tuning the Qwen3.8 27B model in its Q4_K_M GGUF quantization, including a full GGUF/quant stack, a CPU and Vulkan compute backend, the Qwen3.8 forward pass with full attention and GatedDeltaNet, abliteration, and a QLoRA SFT dataset pipeline. The package ships a `qwen-trainer` CLI with `inspect`, `run`, `abliterate`, `search`, `evaluate`, `dataset` and `train` commands, plus a `qwen-server` binary exposing an OpenAI-compatible API with `GET /v1/models`, `POST /v1/chat/completions` and `GET /health`. The server serializes complete forward passes through a Vulkan execution guard so requests interleave between decode steps without corrupting shared command and descriptor state, and GPU kernels from different requests do not execute in parallel.

A pure-Go, cgo-free toolkit for the Qwen3.8 27B model in its Q4 K M GGUF quantization. It implements a complete GGUF/quant stack, a CPU + Vulkan compute backend, the Qwen3.8 forward pass full attention + GatedDeltaNet , abliteration, and a dataset pipeline for SFT Training QLoRA . - Go 1.27 build and tests . - glslc and spirv-val to rebuild the committed SPIR-V shaders. - A Vulkan device for GPU acceleration and the GPU tests the CLI selects it automatically when it has enough memory . - python3 with venv support only for the external verify- tests. 1. Download Qwen3.8-27B Unsloth Dynamic Q4 K M from Hugging Face: - Repository: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF https://huggingface.co/unsloth/Qwen3.8-27B-GGUF - File: in the Files and versions list on the right side of the repo page, download Qwen3.8-27B-UD-Q4 K M.gguf Place it at models/Qwen3.8-27B-UD-Q4 K M.gguf , or let the CLI fetch the pinned artifact with resume + SHA-256 verification for you: ./build/qwen-trainer download; 1. Export the training data from your Opencode environment and use the session history as the teacher model. This writes the manifest.json , sft/<topic /sessions.jsonl , sft/<topic /turns.jsonl , subagents/<topic /... , and rl/metadata.jsonl . opencode-reasoning-extractor ~/.local/share/opencode /my/own/reasoning-traces; 1. Point the Qwen Trainer at that output dataset directory see Usage usage . compile shaders, then build build/qwen-trainer make; the command is always the first argument; flags including --model follow it run the Qwen Trainer ./build/qwen-trainer inspect --model models/Qwen3.8-27B-UD-Q4 K M.gguf; ./build/qwen-trainer run --model models/Qwen3.8-27B-UD-Q4 K M.gguf --prompt "The capital of France is" --generate 8; tokenize/validate the extractor output and report loss-mask statistics --dataset is repeatable: pass it once per exported dataset directory ./build/qwen-trainer dataset --model models/Qwen3.8-27B-UD-Q4 K M.gguf \ --dataset /path/to/my/own/reasoning-traces/from/opencode; --model is required by every model command inspect , run , abliterate , search , evaluate , dataset , train ; download , gpu-info , and status do not need it. There is no implicit default model path anymore. The compute backend is chosen automatically: Vulkan is used when a device is present and it has enough device memory for the model, otherwise the CPU reference runs. gpu-info reports the detected device and memory. See Abliteration abliteration for removing refusal directions and for the parameter search. qwen-server exposes the same Vulkan/CPU inference pipeline through a minimal OpenAI-compatible API. Build it with make server : make server; ./build/qwen-server --model models/Qwen3.8-27B-UD-Q4 K M.gguf --port 8080; It serves GET /v1/models , POST /v1/chat/completions , and GET /health . Up to --max-concurrent requests are admitted at once. Complete forward passes are serialized by the Vulkan execution guard, so requests interleave between decode steps without corrupting shared command/descriptor state. GPU kernels from different requests do not execute in parallel. curl -N http://127.0.0.1:8080/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "Qwen3.8-27B-UD-Q4 K M", "messages": {"role": "user", "content": "What is 2+2?"} , "stream": true }'; Supported request features include stream SSE , reasoning content deltas, tools / tool calls , temperature , top p , top k , seed , stop , max tokens / max completion tokens , reasoning effort , and chat template kwargs enable thinking , preserve thinking . Thinking is enabled by default. Tool calls use the Qwen XML format and are returned as OpenAI tool calls with JSON arguments. Useful flags: | Flag | Values | Default | |---|---|---| | --device | auto , cpu, vulkan | auto | | --context-length | 128k , 256k , 384k , 512k | GGUF context | | --kv-cache-type | q4 k , q8 0 | q4 k | | --max-concurrent | int | 1 | | --default-max-tokens | int | 1024 | | --speculative | true , false | true | | --spec-draft-max | int | 4 | | --spec-draft-min-prob | float64 | 0.6 | | --model-id | string | GGUF filename | | --api-key | string | | Environment variables: | Variable | Description | |---|---| | QWEN SERVER MODEL | Model GGUF Path | | QWEN SERVER API KEY | Bearer API Key | On Vulkan the full inference forward runs device-resident, and K/V are stored in the quantized cache and appended with GPU shaders. The prefill chunk size is selected automatically from the context length and device memory. Streaming usage is included when stream options.include usage is true. MTP speculative decoding is enabled by default on both Vulkan and CPU. Context values above the native GGUF context 256k select YaRN automatically, with the factor derived from context / native context . See ARCHITECTURE.md https://github.com/cookiengineer/qwen-reasoning-trainer/blob/master/docs/ARCHITECTURE.md for the underlying pipeline. Remove the refusal/safety direction from the weights with Heretic-style directional ablation projected / magnitude-preserving orthogonal ablation, MPOA , then optionally optimize the ablation hyperparameters. 1. Ablate with the default parameters and write a full Q4 K M GGUF. ./build/qwen-trainer abliterate --model models/Qwen3.8-27B-UD-Q4 K M.gguf --out models/abliterated.gguf; 2. optional search direction index + per-component weight kernels and write the best trial. Trials persist to the study file and resume automatically. ./build/qwen-trainer search --model models/Qwen3.8-27B-UD-Q4 K M.gguf \ --trials 25 \ --sampler tpe \ --study models/search.jsonl \ --out models/search-best.gguf; 3. Score the result: refusal-keyword rate and KL divergence from the base. ./build/qwen-trainer evaluate --model models/abliterated.gguf \ --prompts source/prompts/harmful behaviors test.jsonl \ --max-tokens 32; source/prompts/ bundles the same datasets Heretic uses, pre-converted to JSON Lines one {"text": "..."} object per line . They are compiled into the binary with go:embed , so abliterate and search use them by default with no path needed; override with --good / --bad --eval-good / --eval-bad for search . Files may be plain text one prompt per line or JSONL --prompt-col selects the column, default text . | File | Rows | Role default | |---|---|---| | harmless alpaca train.jsonl | 25058 | good prompts / residual directions --good | | harmful behaviors train.jsonl | 416 | bad prompts / residual directions --bad | | harmless alpaca test.jsonl | 6265 | KL divergence evaluation --eval-good | | harmful behaviors test.jsonl | 104 | refusal evaluation --eval-bad | The full files are large, so the limits default to Heretic's: --limit 400 residual prompts per set and --eval-limit 100 evaluation prompts per set . Pass --limit 0 / --eval-limit 0 to use an entire file. Every prompt costs one model forward, so raising the limits directly increases runtime. These mirror mlabonne/harmless alpaca and mlabonne/harmful behaviors as published under heretic-org on Hugging Face. | Flag | Command | Meaning | |---|---|---| | --row-norm none\|pre\|full | abliterate , search | MPOA row normalization default full | | --orthogonalize | abliterate , search | project the direction orthogonal to the good direction default true | | --direction-index | abliterate | single global direction layer index -1 = per layer, default | | --winsorize | abliterate , search | clamp residual components to a quantile in 0,1 | | --sampler random\|halton\|tpe | search | optimizer default random | | --lambda | search | KL weight in the objective refusal + lambda KL default 1.0 | | --seed , --study | search | reproducible seed and resumable JSON Lines checkpoint | Note: the first-token KL is a weak signal for a thinking model it emits <think first , so small divergence scores are expected on short generations. evaluate scores a model along three independent axes; at least one input is required. Refusal keyword rate and KL divergence from a base with --base . ./build/qwen-trainer evaluate --model models/abliterated.gguf \ --prompts source/prompts/harmful behaviors test.jsonl --max-tokens 32 \ --base models/Qwen3.8-27B-UD-Q4 K M.gguf; Corpus perplexity over a text file, scored in windows of N tokens. ./build/qwen-trainer evaluate --model models/Qwen3.8-27B-UD-Q4 K M.gguf --perplexity corpus.txt --window 512; Multiple-choice accuracy over JSONL {context, choices, answer} records. ./build/qwen-trainer evaluate --model models/Qwen3.8-27B-UD-Q4 K M.gguf --tasks tasks.jsonl --tasks-verbose; Perplexity uses non-overlapping windows default 512 tokens, --window 0 for the whole input ; the window bounds peak memory at vocab window 4 bytes, so it is the quality/memory knob. Because it computes full LM-head logits for every window it is considerably slower per token than run . Task accuracy is scored by the length-normalized log-likelihood of each choice given the context --tasks-normalize=false for raw sums . The default suite is pure Go and needs neither a GPU nor Python: make test; go test ./... make check; fmt-check + vet + test run this before calling anything done To run the suite on a remote GPU host, stream the working tree over SSH and run a make target there: bash run-tests.sh debby; remote make check bash run-tests.sh debby test; remote make test Tests that need hardware or external data skip automatically, so make check works on any machine: - Vulkan tests skip when no device is present. - Real-model tests tokenizer specials, example building skip when no GGUF is present under models/ . Independent cross-checks live under source/e2e-tests/ https://github.com/cookiengineer/qwen-reasoning-trainer/blob/master/source/e2e-tests , one isolated main program per check. They shell out to reference implementations and are never run by make check : make verify-ref; NumPy forward pass vs the Go tiny model ~3e-8 match make verify-data; Jinja2 render of the model chat template + Python regex evaluation of the qwen35 pre-tokenizer make verify-quants; our block decoders + encoders vs llama.cpp's ggml-quants.c needs a C compiler Each is also runnable directly, e.g. go -C source run ./e2e-tests/verify-ref . The Python references use a project-local virtualenv at source/e2e-tests/.venv , created and populated on demand from each check's own requirements.txt source/e2e-tests/verify-ref/requirements.txt , source/e2e-tests/verify-data/requirements.txt — numpy , jinja2 , regex . The venv is gitignored. - make e2e-venv creates/populates the venv explicitly. - SYSTEM PYTHON=python3.x selects the base interpreter used to build it. - E2E PYTHON=/path/to/python runs the references against your own environment. The programs can also be driven directly with QWEN38 REF PYTHON : QWEN38 REF PYTHON=/path/to/python go -C source run ./e2e-tests/verify-ref QWEN38 REF PYTHON=/path/to/python go -C source run ./e2e-tests/verify-data go -C source run ./e2e-tests/verify-quants CC=... to pick a compiler The dataset reader is additionally checked against a fixture emitted by the opencode-reasoning-extractor 's own writer source/internal/dataset/testdata/extractor-writer/ . Released under the MIT License. See LICENSE.txt https://github.com/cookiengineer/qwen-reasoning-trainer/blob/master/LICENSE.txt .