# Show HN: Qwen Reasoning Traces Trainer and Inference Server

> Source: <https://github.com/cookiengineer/qwen-reasoning-trainer>
> Published: 2026-11-12 06:19:22+00:00

A pure-Go, cgo-free toolkit for the **Qwen3.8 27B** model in its **Q4_K_M** GGUF
quantization. It implements a complete GGUF/quant stack, a CPU + Vulkan compute
backend, the Qwen3.8 forward pass (full attention + GatedDeltaNet), abliteration,
and a dataset pipeline for SFT Training (QLoRA).

- Go 1.27 (build and tests).
- `glslc` and`spirv-val` to rebuild the committed SPIR-V shaders.
- A Vulkan device for GPU acceleration and the GPU tests (the CLI selects it automatically when it has enough memory).
- `python3` with venv support only for the external`verify-*` tests.

1. Download **Qwen3.8-27B** (Unsloth Dynamic**Q4_K_M** ) from Hugging Face:

- Repository: [https://huggingface.co/unsloth/Qwen3.8-27B-GGUF](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF)
- File: in the **Files and versions** list on the right side of the repo page,
download`Qwen3.8-27B-UD-Q4_K_M.gguf`

Place it at `models/Qwen3.8-27B-UD-Q4_K_M.gguf`, or let the CLI fetch the pinned
artifact (with resume + SHA-256 verification) for you:

```
./build/qwen-trainer download;
```

1. Export the training data from your Opencode environment and use the session
history as the teacher model. This writes the `manifest.json` ,`sft/<topic>/sessions.jsonl` ,`sft/<topic>/turns.jsonl` ,`subagents/<topic>/...` ,
and`rl/metadata.jsonl` .

```
opencode-reasoning-extractor ~/.local/share/opencode /my/own/reasoning-traces;
```

1. Point the Qwen Trainer at that output dataset directory (see [Usage](#usage) ).

```
# compile shaders, then build build/qwen-trainer
make;

# the command is always the first argument; flags (including --model) follow it

# run the Qwen Trainer
./build/qwen-trainer inspect --model models/Qwen3.8-27B-UD-Q4_K_M.gguf;
./build/qwen-trainer run --model models/Qwen3.8-27B-UD-Q4_K_M.gguf --prompt "The capital of France is" --generate 8;

# tokenize/validate the extractor output and report loss-mask statistics
# --dataset is repeatable: pass it once per exported dataset directory
./build/qwen-trainer dataset --model models/Qwen3.8-27B-UD-Q4_K_M.gguf \
  --dataset /path/to/my/own/reasoning-traces/from/opencode;
```

`--model` is required by every model command (`inspect`, `run`, `abliterate`,
`search`, `evaluate`, `dataset`, `train`); `download`, `gpu-info`, and `status`
do not need it. There is no implicit default model path anymore.

The compute backend is chosen automatically: Vulkan is used when a device is
present and it has enough device memory for the model, otherwise the CPU
reference runs. `gpu-info` reports the detected device and memory.

See [Abliteration](#abliteration) for removing refusal directions and for the
parameter search.

`qwen-server` exposes the same Vulkan/CPU inference pipeline through a minimal
OpenAI-compatible API. Build it with `make server`:

```
make server;
./build/qwen-server --model models/Qwen3.8-27B-UD-Q4_K_M.gguf --port 8080;
```

It serves `GET /v1/models`, `POST /v1/chat/completions`, and `GET /health`.
Up to `--max-concurrent` requests are admitted at once. Complete forward passes
are serialized by the Vulkan execution guard, so requests interleave between
decode steps without corrupting shared command/descriptor state. GPU kernels
from different requests do not execute in parallel.

```
curl -N http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3.8-27B-UD-Q4_K_M",
    "messages": [{"role": "user", "content": "What is 2+2?"}],
    "stream": true
  }';
```

Supported request features include `stream` (SSE), `reasoning_content` deltas,
`tools`/` tool_calls`, `temperature`, `top_p`, `top_k`, `seed`, `stop`,
`max_tokens`/` max_completion_tokens`, `reasoning_effort`, and
`chat_template_kwargs` (`enable_thinking`, `preserve_thinking`). Thinking is
enabled by default. Tool calls use the Qwen XML format and are returned as
OpenAI `tool_calls` with JSON arguments.

Useful flags:

| Flag | Values | Default | 
|---|---|---|
| `--device` | `auto` ,`cpu,` vulkan` | `auto` | 
| `--context-length` | `128k` ,`256k` ,`384k` ,`512k` | (GGUF context) | 
| `--kv-cache-type` | `q4_k` ,`q8_0` | `q4_k` | 
| `--max-concurrent` | (int) | `1` | 
| `--default-max-tokens` | (int) | `1024` | 
| `--speculative` | `true` ,`false` | `true` | 
| `--spec-draft-max` | (int) | `4` | 
| `--spec-draft-min-prob` | (float64) | `0.6` | 
| `--model-id` | (string) | (GGUF filename) | 
| `--api-key` | (string) |  | 

Environment variables:

| Variable | Description | 
|---|---|
| `QWEN_SERVER_MODEL` | Model GGUF Path | 
| `QWEN_SERVER_API_KEY` | Bearer API Key | 

On Vulkan the full inference forward runs device-resident, and K/V are stored in the quantized cache and appended with GPU shaders. The prefill chunk size is selected automatically from the context length and device memory.

Streaming usage is included when `stream_options.include_usage` is true.

MTP speculative decoding is enabled by default on both Vulkan and CPU.

Context values above the native GGUF context (256k) select YaRN automatically,
with the factor derived from `context / native_context`.

See [ARCHITECTURE.md](https://github.com/cookiengineer/qwen-reasoning-trainer/blob/master/docs/ARCHITECTURE.md) for the underlying pipeline.

Remove the refusal/safety direction from the weights with Heretic-style directional ablation (projected / magnitude-preserving orthogonal ablation, MPOA), then optionally optimize the ablation hyperparameters.

```
# 1. Ablate with the default parameters and write a full Q4_K_M GGUF.
./build/qwen-trainer abliterate --model models/Qwen3.8-27B-UD-Q4_K_M.gguf --out models/abliterated.gguf;

# 2. (optional) search direction index + per-component weight kernels and write
#    the best trial. Trials persist to the study file and resume automatically.
./build/qwen-trainer search --model models/Qwen3.8-27B-UD-Q4_K_M.gguf \
  --trials 25 \
  --sampler tpe \
  --study models/search.jsonl \
  --out models/search-best.gguf;

# 3. Score the result: refusal-keyword rate and KL divergence from the base.
./build/qwen-trainer evaluate --model models/abliterated.gguf \
  --prompts source/prompts/harmful_behaviors_test.jsonl \
  --max-tokens 32;
```

`source/prompts/` bundles the same datasets Heretic uses, pre-converted to JSON
Lines (one `{"text": "..."}` object per line). They are compiled into the binary
with `go:embed`, so `abliterate` and `search` use them by default with no path
needed; override with `--good`/`--bad` (`--eval-good`/`--eval-bad` for `search`).
Files may be plain text (one prompt per line) or JSONL (`--prompt-col` selects
the column, default `text`).

| File | Rows | Role (default) | 
|---|---|---|
| `harmless_alpaca_train.jsonl` | 25058 | good prompts / residual directions ( `--good` ) | 
| `harmful_behaviors_train.jsonl` | 416 | bad prompts / residual directions ( `--bad` ) | 
| `harmless_alpaca_test.jsonl` | 6265 | KL divergence evaluation ( `--eval-good` ) | 
| `harmful_behaviors_test.jsonl` | 104 | refusal evaluation ( `--eval-bad` ) | 

The full files are large, so the limits default to Heretic's: `--limit 400`
(residual prompts per set) and `--eval-limit 100` (evaluation prompts per set).
Pass `--limit 0` / `--eval-limit 0` to use an entire file. Every prompt costs one
model forward, so raising the limits directly increases runtime.

These mirror `mlabonne/harmless_alpaca` and `mlabonne/harmful_behaviors` as
published under `heretic-org` on Hugging Face.

| Flag | Command | Meaning | 
|---|---|---|
| `--row-norm none\|pre\|full` | `abliterate` ,`search` | MPOA row normalization (default `full` ) | 
| `--orthogonalize` | `abliterate` ,`search` | project the direction orthogonal to the good direction (default true) | 
| `--direction-index` | `abliterate` | single global direction layer index ( `-1` = per layer, default) | 
| `--winsorize` | `abliterate` ,`search` | clamp residual components to a quantile in `[0,1)` | 
| `--sampler random\|halton\|tpe` | `search` | optimizer (default `random` ) | 
| `--lambda` | `search` | KL weight in the objective `refusal + lambda*KL` (default`1.0` ) | 
| `--seed` ,`--study` | `search` | reproducible seed and resumable JSON Lines checkpoint | 

Note: the first-token KL is a weak signal for a thinking model (it emits
`<think>` first), so small divergence scores are expected on short generations.

`evaluate` scores a model along three independent axes; at least one input is
required.

```
# Refusal keyword rate (and KL divergence from a base with --base).
./build/qwen-trainer evaluate --model models/abliterated.gguf \
  --prompts source/prompts/harmful_behaviors_test.jsonl --max-tokens 32 \
  --base models/Qwen3.8-27B-UD-Q4_K_M.gguf;

# Corpus perplexity over a text file, scored in windows of N tokens.
./build/qwen-trainer evaluate --model models/Qwen3.8-27B-UD-Q4_K_M.gguf --perplexity corpus.txt --window 512;

# Multiple-choice accuracy over JSONL {context, choices, answer} records.
./build/qwen-trainer evaluate --model models/Qwen3.8-27B-UD-Q4_K_M.gguf --tasks tasks.jsonl --tasks-verbose;
```

Perplexity uses non-overlapping windows (default 512 tokens, `--window 0` for
the whole input); the window bounds peak memory at `vocab * window * 4` bytes,
so it is the quality/memory knob. Because it computes full LM-head logits for
every window it is considerably slower per token than `run`. Task accuracy is
scored by the length-normalized log-likelihood of each choice given the context
(`--tasks-normalize=false` for raw sums).

The default suite is pure Go and needs neither a GPU nor Python:

```
make test;      # go test ./...
make check;     # fmt-check + vet + test (run this before calling anything done)
```

To run the suite on a remote GPU host, stream the working tree over SSH and run a make target there:

```
bash run-tests.sh debby;          # remote make check
bash run-tests.sh debby test;     # remote make test
```

Tests that need hardware or external data skip automatically, so `make check`
works on any machine:

- Vulkan tests skip when no device is present.
- Real-model tests (tokenizer specials, example building) skip when no GGUF is
present under `models/` .

Independent cross-checks live under [source/e2e-tests/](https://github.com/cookiengineer/qwen-reasoning-trainer/blob/master/source/e2e-tests), one
isolated `main` program per check. They shell out to reference implementations
and are never run by `make check`:

```
make verify-ref;    # NumPy forward pass vs the Go tiny model (~3e-8 match)
make verify-data;   # Jinja2 render of the model chat template + Python `regex`
                    # evaluation of the qwen35 pre-tokenizer
make verify-quants; # our block decoders + encoders vs llama.cpp's ggml-quants.c
                    # (needs a C compiler)
```

Each is also runnable directly, e.g. `go -C source run ./e2e-tests/verify-ref`.

The Python references use a project-local virtualenv at `source/e2e-tests/.venv`,
created and populated on demand from each check's own `requirements.txt`
(`source/e2e-tests/verify-ref/requirements.txt`,
`source/e2e-tests/verify-data/requirements.txt` — `numpy`, `jinja2`, `regex`).
The venv is gitignored.

- `make e2e-venv` creates/populates the venv explicitly.
- `SYSTEM_PYTHON=python3.x` selects the base interpreter used to build it.
- `E2E_PYTHON=/path/to/python` runs the references against your own environment.

The programs can also be driven directly with `QWEN38_REF_PYTHON`:

```
QWEN38_REF_PYTHON=/path/to/python go -C source run ./e2e-tests/verify-ref
QWEN38_REF_PYTHON=/path/to/python go -C source run ./e2e-tests/verify-data
go -C source run ./e2e-tests/verify-quants     # CC=... to pick a compiler
```

The dataset reader is additionally checked against a fixture emitted by the
`opencode-reasoning-extractor`'s own writer
(`source/internal/dataset/testdata/extractor-writer/`).

Released under the MIT License. See [LICENSE.txt](https://github.com/cookiengineer/qwen-reasoning-trainer/blob/master/LICENSE.txt).