Show HN: Qwen Reasoning Traces Trainer and Inference Server

A pure-Go, cgo-free toolkit for the Qwen3.8 27B model in its Q4_K_M GGUF quantization. It implements a complete GGUF/quant stack, a CPU + Vulkan compute backend, the Qwen3.8 forward pass (full attention + GatedDeltaNet), abliteration, and a dataset pipeline for SFT Training (QLoRA).

• Go 1.27 (build and tests).
• glslc andspirv-val to rebuild the committed SPIR-V shaders.
• A Vulkan device for GPU acceleration and the GPU tests (the CLI selects it automatically when it has enough memory).
• python3 with venv support only for the externalverify-* tests.

1. Download Qwen3.8-27B (Unsloth DynamicQ4_K_M ) from Hugging Face:

• Repository: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
• File: in the Files and versions list on the right side of the repo page, downloadQwen3.8-27B-UD-Q4_K_M.gguf

Place it at models/Qwen3.8-27B-UD-Q4_K_M.gguf, or let the CLI fetch the pinned artifact (with resume + SHA-256 verification) for you:

./build/qwen-trainer download;

1. Export the training data from your Opencode environment and use the session history as the teacher model. This writes the manifest.json ,sft/<topic>/sessions.jsonl ,sft/<topic>/turns.jsonl ,subagents/<topic>/... , andrl/metadata.jsonl .

opencode-reasoning-extractor ~/.local/share/opencode /my/own/reasoning-traces;

1. Point the Qwen Trainer at that output dataset directory (see Usage ).

make;


./build/qwen-trainer inspect --model models/Qwen3.8-27B-UD-Q4_K_M.gguf;
./build/qwen-trainer run --model models/Qwen3.8-27B-UD-Q4_K_M.gguf --prompt "The capital of France is" --generate 8;

./build/qwen-trainer dataset --model models/Qwen3.8-27B-UD-Q4_K_M.gguf \
  --dataset /path/to/my/own/reasoning-traces/from/opencode;

--model is required by every model command (inspect, run, abliterate, search, evaluate, dataset, train); download, gpu-info, and status do not need it. There is no implicit default model path anymore.

The compute backend is chosen automatically: Vulkan is used when a device is present and it has enough device memory for the model, otherwise the CPU reference runs. gpu-info reports the detected device and memory.

See Abliteration for removing refusal directions and for the parameter search.

qwen-server exposes the same Vulkan/CPU inference pipeline through a minimal OpenAI-compatible API. Build it with make server:

make server;
./build/qwen-server --model models/Qwen3.8-27B-UD-Q4_K_M.gguf --port 8080;

It serves GET /v1/models, POST /v1/chat/completions, and GET /health. Up to --max-concurrent requests are admitted at once. Complete forward passes are serialized by the Vulkan execution guard, so requests interleave between decode steps without corrupting shared command/descriptor state. GPU kernels from different requests do not execute in parallel.

curl -N http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3.8-27B-UD-Q4_K_M",
    "messages": [{"role": "user", "content": "What is 2+2?"}],
    "stream": true
  }';

Supported request features include stream (SSE), reasoning_content deltas, tools/ tool_calls, temperature, top_p, top_k, seed, stop, max_tokens/ max_completion_tokens, reasoning_effort, and chat_template_kwargs (enable_thinking, preserve_thinking). Thinking is enabled by default. Tool calls use the Qwen XML format and are returned as OpenAI tool_calls with JSON arguments.

Useful flags:

Flag	Values	Default
--device	auto ,cpu, vulkan`	auto
--context-length	128k ,256k ,384k ,512k	(GGUF context)
--kv-cache-type	q4_k ,q8_0	q4_k
--max-concurrent	(int)	1
--default-max-tokens	(int)	1024
--speculative	true ,false	true
--spec-draft-max	(int)	4
--spec-draft-min-prob	(float64)	0.6
--model-id	(string)	(GGUF filename)
--api-key	(string)

Environment variables:

Variable	Description
QWEN_SERVER_MODEL	Model GGUF Path
QWEN_SERVER_API_KEY	Bearer API Key

On Vulkan the full inference forward runs device-resident, and K/V are stored in the quantized cache and appended with GPU shaders. The prefill chunk size is selected automatically from the context length and device memory.

Streaming usage is included when stream_options.include_usage is true.

MTP speculative decoding is enabled by default on both Vulkan and CPU.

Context values above the native GGUF context (256k) select YaRN automatically, with the factor derived from context / native_context.

See ARCHITECTURE.md for the underlying pipeline.

Remove the refusal/safety direction from the weights with Heretic-style directional ablation (projected / magnitude-preserving orthogonal ablation, MPOA), then optionally optimize the ablation hyperparameters.

./build/qwen-trainer abliterate --model models/Qwen3.8-27B-UD-Q4_K_M.gguf --out models/abliterated.gguf;

./build/qwen-trainer search --model models/Qwen3.8-27B-UD-Q4_K_M.gguf \
  --trials 25 \
  --sampler tpe \
  --study models/search.jsonl \
  --out models/search-best.gguf;

./build/qwen-trainer evaluate --model models/abliterated.gguf \
  --prompts source/prompts/harmful_behaviors_test.jsonl \
  --max-tokens 32;

source/prompts/ bundles the same datasets Heretic uses, pre-converted to JSON Lines (one {"text": "..."} object per line). They are compiled into the binary with go:embed, so abliterate and search use them by default with no path needed; override with --good/--bad (--eval-good/--eval-bad for search). Files may be plain text (one prompt per line) or JSONL (--prompt-col selects the column, default text).

File	Rows	Role (default)
harmless_alpaca_train.jsonl	25058	good prompts / residual directions ( --good )
harmful_behaviors_train.jsonl	416	bad prompts / residual directions ( --bad )
harmless_alpaca_test.jsonl	6265	KL divergence evaluation ( --eval-good )
harmful_behaviors_test.jsonl	104	refusal evaluation ( --eval-bad )

The full files are large, so the limits default to Heretic's: --limit 400 (residual prompts per set) and --eval-limit 100 (evaluation prompts per set). Pass --limit 0 / --eval-limit 0 to use an entire file. Every prompt costs one model forward, so raising the limits directly increases runtime.

These mirror mlabonne/harmless_alpaca and mlabonne/harmful_behaviors as published under heretic-org on Hugging Face.

Flag	Command	Meaning
--row-norm none|pre|full	abliterate ,search	MPOA row normalization (default full )
--orthogonalize	abliterate ,search	project the direction orthogonal to the good direction (default true)
--direction-index	abliterate	single global direction layer index ( -1 = per layer, default)
--winsorize	abliterate ,search	clamp residual components to a quantile in [0,1)
--sampler random|halton|tpe	search	optimizer (default random )
--lambda	search	KL weight in the objective refusal + lambda*KL (default1.0 )
--seed ,--study	search	reproducible seed and resumable JSON Lines checkpoint

Note: the first-token KL is a weak signal for a thinking model (it emits <think> first), so small divergence scores are expected on short generations.

evaluate scores a model along three independent axes; at least one input is required.

./build/qwen-trainer evaluate --model models/abliterated.gguf \
  --prompts source/prompts/harmful_behaviors_test.jsonl --max-tokens 32 \
  --base models/Qwen3.8-27B-UD-Q4_K_M.gguf;

./build/qwen-trainer evaluate --model models/Qwen3.8-27B-UD-Q4_K_M.gguf --perplexity corpus.txt --window 512;

./build/qwen-trainer evaluate --model models/Qwen3.8-27B-UD-Q4_K_M.gguf --tasks tasks.jsonl --tasks-verbose;

Perplexity uses non-overlapping windows (default 512 tokens, --window 0 for the whole input); the window bounds peak memory at vocab * window * 4 bytes, so it is the quality/memory knob. Because it computes full LM-head logits for every window it is considerably slower per token than run. Task accuracy is scored by the length-normalized log-likelihood of each choice given the context (--tasks-normalize=false for raw sums).

The default suite is pure Go and needs neither a GPU nor Python:

make test;      # go test ./...
make check;     # fmt-check + vet + test (run this before calling anything done)

To run the suite on a remote GPU host, stream the working tree over SSH and run a make target there:

bash run-tests.sh debby;          # remote make check
bash run-tests.sh debby test;     # remote make test

Tests that need hardware or external data skip automatically, so make check works on any machine:

• Vulkan tests skip when no device is present.
• Real-model tests (tokenizer specials, example building) skip when no GGUF is present under models/ .

Independent cross-checks live under source/e2e-tests/, one isolated main program per check. They shell out to reference implementations and are never run by make check:

make verify-ref;    # NumPy forward pass vs the Go tiny model (~3e-8 match)
make verify-data;   # Jinja2 render of the model chat template + Python `regex`
make verify-quants; # our block decoders + encoders vs llama.cpp's ggml-quants.c

Each is also runnable directly, e.g. go -C source run ./e2e-tests/verify-ref.

The Python references use a project-local virtualenv at source/e2e-tests/.venv, created and populated on demand from each check's own requirements.txt (source/e2e-tests/verify-ref/requirements.txt, source/e2e-tests/verify-data/requirements.txt — numpy, jinja2, regex). The venv is gitignored.

• make e2e-venv creates/populates the venv explicitly.
• SYSTEM_PYTHON=python3.x selects the base interpreter used to build it.
• E2E_PYTHON=/path/to/python runs the references against your own environment.

The programs can also be driven directly with QWEN38_REF_PYTHON:

QWEN38_REF_PYTHON=/path/to/python go -C source run ./e2e-tests/verify-ref
QWEN38_REF_PYTHON=/path/to/python go -C source run ./e2e-tests/verify-data
go -C source run ./e2e-tests/verify-quants     # CC=... to pick a compiler

The dataset reader is additionally checked against a fixture emitted by the opencode-reasoning-extractor's own writer (source/internal/dataset/testdata/extractor-writer/).

Released under the MIT License. See LICENSE.txt.

source & further reading

github.com — original article Show HN: JNum – Pure Java 25 NumPy/BLAS via Project Panama FFM and Vector API Open DLSS alternative claims better performance and quality with AMD support Show HN: Unglazed – paper-to-explainer-video tool that gets every detail