{"slug": "xyntetik-runner-a-gguf-runtime-in-plain-c-that-proves-what-it-runs", "title": "Xyntetik Runner – a GGUF runtime in plain C that proves what it runs", "summary": "Xyntetik Runner, a GGUF model runtime written from scratch in plain C, is now available as a free Apache 2.0-licensed engine that serves, verifies, scores, adapts, and trains GGUF models deterministically, with support for CPU (x86 AVX2/FMA, ARM NEON), CUDA, and Metal. The project, built in Sweden and bootstrapped by consulting and enterprise work, emphasizes verifiable claims and compatibility with llama.cpp formats, including adapters that score identically (1.000 on held-out eval) when served by stock llama.cpp.", "body_md": "One binary is the whole model runtime: it **serves, verifies, scores,\nadapts and trains** GGUF models - deterministically, with every claim tied\nto a measurement you can re-run. Written from scratch in plain C. No\nPython, no pip, no third-party runtime, no ggml. CPU (x86 AVX2/FMA, ARM\nNEON), CUDA, and Metal.\n\n```\ncurl -LO https://github.com/Joakimpalm-Zen/xyntetik-runner/releases/latest/download/runner-macos-arm64\ncurl -LO https://github.com/Joakimpalm-Zen/xyntetik-runner/releases/latest/download/SHA256SUMS\nshasum -a 256 --check --ignore-missing SHA256SUMS\nchmod +x runner-macos-arm64 && mv runner-macos-arm64 runner\ncurl -L -o model.gguf https://huggingface.co/ibm-granite/granite-4.1-3b-GGUF/resolve/main/granite-4.1-3b-Q8_0.gguf\n./runner -m model.gguf --serve\ncurl localhost:8080/v1/chat/completions \\\n  -d '{\"messages\":[{\"role\":\"user\",\"content\":\"Say hello in one sentence.\"}]}'\n```\n\nLinux: the asset is `runner-linux-x86_64`\n\nand the check command is\n`sha256sum -c --ignore-missing`\n\n. Windows: `runner-windows-x86_64.exe`\n\n.\nThe checksum line is not decoration - this project's whole culture is\nreceipts, and it starts at the download. The model above is the smallest\nthat passes this project's fidelity gate against its own BF16 parent;\nalternatives and the reasoning are in the [quick start](#build-from-source)\nbelow.\n\nmacOS note: the binaries are not yet notarized. A `curl`\n\ndownload runs as\nshown; a *browser* download gets quarantined by Gatekeeper - clear it with\n`xattr -d com.apple.quarantine runner`\n\nor right-click → Open once.\n\n**Testing Runner?** The project is pre-1.0 and hardware coverage is still\nlimited - that is an invitation, not an apology. If you have an\nNVIDIA/Apple setup, an unusual GGUF, a coding agent, or a model family\nnot in the [support matrix](#support-matrix), the result is genuinely\nwanted, success or failure alike:\n[open an issue](/Joakimpalm-Zen/xyntetik-runner/issues) with `runner --version`\n\n, `runner --caps`\n\n,\nthe model's exact filename and the load log. Independent reproductions\nof the determinism claims get credited in the docs, as the first one\nalready is.\n\n**Contents:** [try it](#sixty-seconds-to-a-served-model) ·\n[why Runner](#why-this-and-not-llamacpp) ·\n[what it adds](#what-runner-adds) · [training](#adaptation) ·\n[models](#models-and-conversion) · [APIs](#serving-and-apis) ·\n[support matrix](#support-matrix) ·\n[CLI reference](#command-line-reference)\n\nXyntetik Runner is independent and bootstrapped: **the engine is free\nforever under Apache 2.0** - consulting and enterprise work fund the\nhardware. Built in Sweden, runs on your hardware; your data never leaves\nthe building.\n\nUse llama.cpp - it is the ecosystem, and Runner deliberately rides its\nformats rather than competing with them: GGUF in, llama.cpp-convention\nadapter files in *and* out. An adapter Runner trains scores identically\n(1.000 on its held-out eval) served by stock llama.cpp, and community F16\nadapters load straight back into Runner - both measured, not assumed.\n\nWhat Runner adds is not a longer feature list; it is a set of\n**contracts** the ecosystem does not make. Determinism as a hard promise:\nthe same executable and inputs reproduce the same sampled tokens across\nruns and thread counts, and training reproduces the same adapter file\nsha256, gated in CI. Independent rebuilds are explicitly outside that\nbyte-identity claim because compiler and ISA libm can differ. Scope as a\npromise: supported architectures are\nnamed, unknown ones are refused, and every backend claim is tied to an\nexecutable gate and pinned model evidence. Honesty as an artifact: the\nbenchmark tables below include the rows where Runner loses, and the docs\nkeep the failed experiments. If you want maximum architecture coverage and\nraw speed, use llama.cpp and we mean that sincerely. If you need to prove\nwhat your model said, what it learned from, or what you actually shipped -\nthat is what this runtime is for.\n\nFor release history and benchmark narratives, see [CHANGELOG.md](/Joakimpalm-Zen/xyntetik-runner/blob/main/CHANGELOG.md)\nand [docs/benchmarks.md](/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/benchmarks.md).\n\nDownload a prebuilt binary from the [latest release](/Joakimpalm-Zen/xyntetik-runner/releases/latest)\nfor Linux, macOS, or Windows, or build from source:\n\n```\ngit clone https://github.com/Joakimpalm-Zen/xyntetik-runner\ncd xyntetik-runner\nmake\n./runner --version   # -> runner 0.3.0\n```\n\nCUDA builds and releases need only an NVIDIA driver at runtime. The CUDA toolkit is needed only by developers regenerating the embedded PTX.\n\nGPU driver requirement - raised.GPU execution now requires an NVIDIA driver withCUDA 13.0 support or newer(the R580 driver series). The embedded PTX is generated by the CUDA 13.0 toolchain (PTX ISA 9.0) to add the BF16 and Q2_K device kernels; older drivers - the previous floor was the CUDA ~11.8 era - will fail to JIT it, and the runner then reports the failure andfalls back to CPUrather than computing wrong. CPU-only execution is unaffected. Check your driver's CUDA level with`nvidia-smi`\n\n(top-right \"CUDA Version\").\n\nRelease archives name the binary for their platform - `runner-macos-arm64`\n\n,\n`runner-linux-x86_64`\n\n, `runner-windows-x86_64.exe`\n\n- so either rename it to\n`runner`\n\nor substitute that name in the commands below. A source build produces\n`runner`\n\ndirectly.\n\nIf you have no GGUF handy, the measured recommendation at 8 GB is an 8-bit small model, not a 4-bit larger one. granite-4.1-3b Q8_0 (3.6 GB, first-party IBM file) is the smallest model that passes this project's fidelity gate against its own BF16 parent (100% margin-qualified top-1 / 0.0024 mean KLD, 2026-08-14; every 4- and 5-bit quant measured to date fails on distributional distance):\n\n```\ncurl -L -o model.gguf \\\n  https://huggingface.co/ibm-granite/granite-4.1-3b-GGUF/resolve/main/granite-4.1-3b-Q8_0.gguf\n```\n\nFor the fastest possible smoke test on a small machine there is also a 2.63 GB option - know what it is: measured against its own BF16 parent it agrees on 77.75% of tokens (mean KLD 0.286), a try-the-runner artifact, not a faithful gemma-4-E2B; its card carries the full numbers.\n\n```\ncurl -L -o model.gguf \\\n  https://huggingface.co/Joakimpalm-Zen/gemma-4-E2B-it-Q4_0-GGUF/resolve/main/gemma-4-E2B-it-Q4_K_M-Q4_0-mix.gguf\n```\n\nRun a GGUF:\n\n```\n./runner -m model.gguf -i\n./runner -m model.gguf -p \"Explain prefix caching\" --temp 0\n./runner -m model.gguf --serve --parallel 2\n./runner -m model.gguf -p \"Return a status object\" --json\n./runner -m model.gguf -f big-document.txt -c 8192 -n 200\n./runner -m big.gguf --draft small.gguf -p \"Continue this code\"\n```\n\nPre-1.0 (APIs, model coverage and certification envelopes may change between releases. CI builds and smoke-tests Linux, macOS, and Windows, but the project still has limited hardware coverage. Include`0.3.0`\n\n).`runner --version`\n\n,`runner --caps`\n\n, the model's exact filename, and the load log in issue reports. Read[SECURITY.md]for the threat model and[CONTRIBUTING.md]for the required correctness gates.\n\nThe contracts above, made concrete. The two capabilities that matter most have their own sections below; the rest follow as a list, ordered by how much difference each makes in practice.\n\nWhen a tool call runs past its token budget, most engines return an empty or\nmalformed `tool_calls`\n\n- commonly `finish_reason: \"length\"`\n\nwith nothing\nusable, or truncated JSON the caller cannot parse and has to repair or retry.\nRunner closes the call to the smallest schema-legal document instead, so the\narguments still parse. This is **forced-truncation recovery**, not ordinary\nJSON-Schema constrained decoding: once a document starts, Runner emits a legal\nending when the budget expires. On local models, where context is tight and\ngeneration is slow, it is the difference between an agent loop that finishes and\none that retries from scratch - burning tokens, time, and context window.\n\nWhat each engine hands the caller when the token budget cuts a tool call short\n\n- same box, same tool schema, same prompt,\n`tool_choice:\"required\"`\n\n, temperature 0, budgets 1→64:\n\n| engine | budget too small (1–16 tokens) | enough budget (64, control) |\n|---|---|---|\nRunner |\nexecutable `tool_calls` , arguments parse |\ncompletes |\n| vLLM 0.27.1 | no call; protocol framing leaks into `content` |\ncompletes |\n| llama.cpp b10488 | no call; leak, then `tool_calls` with unparseable args |\ncompletes |\n| Ollama 0.32.14 | no call; empty content, then HTTP 500 | completes |\n| TensorRT-LLM 1.2.1 † | no call; `<tool_call>` leak, then empty content |\ncompletes |\n| SGLang 0.5.17 † | no call; `<tool_call>` leak, then empty content |\ncompletes |\n\nThe control rung proves the failure is truncation, not misconfiguration: every\nengine completes at 64. Below that, only Runner returns an executable call; the\nothers each hand back something broken or absent. This is the behaviour across\nevery OpenAI-compatible engine we have measured - not a claim about engines we\nhave not. **†** TensorRT-LLM and SGLang were measured on a Qwen3-1.7B substitute\n(their model registries did not carry the granite-4.1-3b used for the other\nfour); truncation recovery is a property of the runtime, so this measures the\nengine, not the model.\n\nThe [truncation benchmark](/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/truncation-benchmark.md) has the full recipe and\nraw responses and pins Runner's column as a per-release regression gate\n(`make test-truncation`\n\n); the [agent-torture gate](/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/agent-torture.md) tests\nthe same failure mode. Tool-call fidelity under **quantization** is measured\ntoo: on a full quant ladder, constrained decoding held schema conformance and\ntool selection at 100% down to Q4_0 while argument agreement decayed to 50% - it\nguarantees the SHAPE of a call at any quantization, not its contents\n([docs/quant-fidelity.md](/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/quant-fidelity.md)).\n\nRunner trains LoRA adapters **directly through the frozen quantized GGUF\nused for inference**. There is no FP16 training copy and no separate\ntraining framework: the serving forward pass is the training forward pass,\nso **the policy you sample is the policy you train** - the train/infer\nnumerical mismatch that silently breaks on-policy learning cannot occur\nbetween two codepaths that are one codepath. And training is deterministic\nin the strongest sense: same data + same seed + same config produce a\n**byte-identical adapter file**, with a machine-written provenance record\n(base/data/adapter sha256s, seed, full config) beside every adapter -\nadaptation as an auditable artifact, not a run that is merely repeatable\n\"within tolerance.\"\n\nMeasured, on a public artifact you can download and reproduce\n([Qwen3-4B-Runner-ToolUse-Q4_K_M](https://huggingface.co/Joakimpalm-Zen/Qwen3-4B-Runner-ToolUse-Q4_K_M)):\n\n| measured result | |\n|---|---|\n| base | Qwen3-4B Q4_K_M (frozen 4-bit serving weights) |\n| training path | directly through the quantized inference artifact, CPU |\n| held-out tool-calling, exact call | 0.69 → 1.00 |\n| reproducibility | two independent runs → byte-identical adapter (same sha256) |\n| precision study | adapters trained through BF16 vs Q8_0: cosine 0.9998; through Q4_K_M: 0.9926 - measurably different objects, capability-equivalent on this task |\n| neutral-corpus drift | nll/token 4.063 → 4.026 (the adapter leaves unrelated text alone) |\n| merge study | `--merge-lora` into Q8_0/F16 keeps the 1.00 (verified in stock llama.cpp); merging into the 4-bit base erases the fine-tune - 0.69 again, 98.55% of weight bytes round back to the base's codes. Scale sweep: survival is monotone in delta magnitude (erased through 2×, partial at 4×, full at 8× - where the exact 8× adapter breaks the served model, the 4-bit grid filters it back to 1.00) |\n| interop | the adapter scores the same 1.00 served by stock llama.cpp; community F16 adapters load back into runner (measured on a third-party adapter, which also found and fixed the F32-only loader gap) |\n\n`--score`\n\ngives teacher-forced logprobs for evals and rewards, `--lora`\n\nserves any adapter back, `--merge-lora`\n\nfolds an adapter into the base for\na standalone GGUF any runtime can serve (with its own provenance record -\nand the honest caveat that a quantized merge rounds the delta; `--lora`\n\nis\nthe exact form), and `scripts/train-grpo-lite.py`\n\ncloses the loop into\nseeded, replayable reinforcement fine-tuning. Design, gates, failure\nmodes and every number above: [docs/adaptation-engine.md](/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/adaptation-engine.md).\n\nThe rest of what sets Runner apart, ordered by how much difference each makes:\n\n**A shared GPU stops being first-come, first-crash.** Run a coding agent beside an embeddings model beside a draft model and the usual outcome is that one load kills another. Runner processes on the same GPU share a VRAM registry: a refused load names every live holder by PID, model, bytes, and uptime,`--wait-for-vram`\n\nturns that refusal into a bounded queue, and records left by dead processes are reaped. It makes a GPU something you can schedule rather than something you hope fits.**You can ask what fits before loading anything.** The usual way to find out whether a model fits is to load it and wait for the failure.`--caps`\n\nneeds no model file and returns one JSON document containing live RAM/VRAM, backend and GPU limits, CPU and GPU quant lists, admitted architectures, placement modes, and model-count limits. A supervisor, tray controller, or CI job can reject an incompatible placement before dispatch, which removes a whole class of load-wait-fail-retry loops. For a specific file,`--fit`\n\nanswers the same question from the model's GGUF header - the first few megabytes - so a ranged read decides whether the rest of the download is worth starting.**Constrained decisions come with a confidence signal.**`choice_logprobs`\n\nrecords each JSON-schema branch as legal alternatives, a posterior renormalized over them, and the probed probability mass - how confident the model was choosing one branch over another, which is what routing and calibrated classification actually need. The included calibration tool turns labeled decisions into accuracy, Brier-score, and ECE gates. This is a decision record rather than ordinary token logprobs, and a power-user feature: most workloads will never reach for it.**A hardware switch has a correctness contract.** If you move a workload between backends and the output quietly changes, that is a bug, not a tuning artifact. CPU/GPU identity here belongs to an exact SHA-256-pinned model and execution path, and faster kernels that reassociate floating-point sums must pass numerical tolerance gates rather than inherit a correctness claim from the backend name. Most users never compare outputs across backends; this is documented because the project treats correctness as a gate, not because it is a headline.\n\nThe full compatibility method is in\n[docs/compatibility-program.md](/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/compatibility-program.md) and performance\nmeasurements are in [docs/performance.md](/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/performance.md). Work that was\nbuilt, measured and rejected is kept too, so it is not attempted twice:\n[docs/negative-result-expert-cache.md](/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/negative-result-expert-cache.md)\nfor MoE expert caching, and\n[docs/negative-result-metal-multirow-matvec.md](/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/negative-result-metal-multirow-matvec.md)\nfor the multi-row Metal decode matvec - which also records what the\nCPU/GPU byte-identity contract costs in reachable GPU optimizations.\n\n```\nmake          # release build: ./runner or runner.exe\nmake debug    # ASan/UBSan development build where supported\nmake test     # unit, fixture, generated-source, and backend gates\n```\n\nRunner uses ordinary platform C, math, threading, mmap/file-mapping, and dynamic-loader libraries. GGUF is little-endian, so little-endian hosts are required.\n\n| Platform | Toolchain | Accelerated path |\n|---|---|---|\n| Linux x86_64 | GCC | AVX2/FMA; CUDA on NVIDIA Turing / compute capability 7.5 or newer, driver with CUDA 13.0+ support (R580 series) |\n| macOS arm64 | Apple Clang | ARM NEON; Metal on Apple Silicon |\n| Windows x86_64 | MinGW-w64 via MSYS2 | AVX2/FMA; CUDA on NVIDIA Turing / compute capability 7.5 or newer, driver with CUDA 13.0+ support (R580 series) |\n\nOn Windows, install `make`\n\nand `mingw-w64-ucrt-x86_64-gcc`\n\nfrom an MSYS2 UCRT64\nshell, then run `make`\n\n.\n\nEach release publishes a CPU image - the same binary on a distroless glibc base,\nnothing else - to `ghcr.io/joakimpalm-zen/xyntetik-runner:<version>`\n\n(and\n`:latest`\n\n). Build it yourself with `docker build -t runner .`\n\n.\n\nThe server binds **loopback only** by design (there is no `--host`\n\n/`0.0.0.0`\n\nflag), so it never exposes itself to a network, even in a container - which\nshapes how you run it:\n\n```\n# One-shot inference (no networking):\ndocker run --rm -v \"$PWD/models:/models\" \\\n  ghcr.io/joakimpalm-zen/xyntetik-runner:latest \\\n  -m /models/your.gguf -p \"hello\" -n 128 --gpu off\n\n# Serve on the host's localhost (Linux; --network host shares the host loopback,\n# so the loopback-only server is reachable at 127.0.0.1:8080 on the host only):\ndocker run --rm --network host -v \"$PWD/models:/models\" \\\n  ghcr.io/joakimpalm-zen/xyntetik-runner:latest \\\n  -m /models/your.gguf --serve --port 8080\n```\n\n`-p 8080:8080`\n\ndoes not work - the port-proxy cannot reach a server bound to the\ncontainer's own loopback; use `--network host`\n\n. There is no auth boundary, so\nkeep any deployment on a trusted host.\n\nThe image is **CPU by default, but GPU-capable without a separate variant.** The\nbinary loads the CUDA **driver** at runtime (`libcuda.so.1`\n\n, the driver API) and\ncarries its kernels as embedded PTX, so it needs no CUDA toolkit baked in - run\nit on an NVIDIA host with the NVIDIA Container Toolkit and `--gpus all`\n\nand the\nrunner uses the GPU:\n\n```\ndocker run --rm --gpus all --network host -v \"$PWD/models:/models\" \\\n  ghcr.io/joakimpalm-zen/xyntetik-runner:latest \\\n  -m /models/your.gguf --serve --port 8080\n```\n\n(A `nvidia/cuda`\n\n-based image is deliberately not published - it would only add a\nCUDA runtime the driver-API path never calls.) Verified on an RTX 3070 via WSL2\n(2026-08-19, Docker 29.1.3 + NVIDIA Container Toolkit 1.19.1): the same\ndistroless image run with `--gpus all`\n\nreports `\"gpu\":{\"backend\":\"cuda\",\"name\": \"NVIDIA GeForce RTX 3070\",…}`\n\nfrom `runner --caps`\n\nand prints `gpu: CUDA backend on NVIDIA GeForce RTX 3070`\n\nwith VRAM accounting at load; the identical image run\nwithout `--gpus`\n\nreports `\"gpu\":null`\n\nand runs on CPU (`libcuda.so.1`\n\nabsent), so\nthe flag is what makes the difference. A generation attempt in that case says\nthat the CUDA driver library is unavailable before continuing on the CPU;\nruntime/device discovery and backend staging failures likewise name the failed\nstage instead of looking like a successful GPU admission. CUDA shared-weight\nsetup also identifies the tensor upload or per-layer table/field that failed,\nincluding architecture-specific recurrent, sink, MoE-bias, and Gemma tables.\nMetal cannot be containerized\n(Apple-Silicon only, no passthrough).\n\nRunner accepts GGUF v2/v3. Safetensors checkpoints must be converted to GGUF\nfirst. Standard llama.cpp multi-part sets (`<prefix>-00001-of-000NN.gguf`\n\n) load\nnatively from any part: every part must be present in the same directory, and\nits `split.no`\n\n, `split.count`\n\n, and `split.tensors.count`\n\nmetadata must agree.\nMissing or inconsistent parts are refused before model binding. Nonstandard\nfilenames and remote/streamed parts are not resolved automatically; merge or\nrename those sets to the standard layout first.\n\nFetch the small test model with:\n\n```\n./download-model.sh\n```\n\nFor manual downloads, verify both the command exit status and resulting byte size. A partially downloaded GGUF can otherwise look like a model failure.\n\nRepack weight matrices to `q8_0`\n\n, `q4_0`\n\n, or `f16`\n\n:\n\n```\n./runner -m model-f16.gguf --quantize model-q4.gguf --quant q4_0\n```\n\nNorms, biases, and rope factors stay f32; tensors already smaller than the\ntarget are retained, as are rows the target type cannot describe (`q3_k`\n\nneeds\na row width divisible by 256, `q8_0`\n\n/`q4_0`\n\nby 32). MoE router weights\n(`ffn_gate_inp*`\n\n) keep their source type on every path, including a\n`--type-plan`\n\nthat names them: the router selects which expert runs, so an\nerror there swaps a whole FFN, and it is a fraction of a percent of the file.\nMetadata is copied.\n\nA `q4_0`\n\nrepack is **lossless where the source is already on the q4_0 grid**,\nwhich is the case for quantization-aware-trained checkpoints: every value is\none per-block scale times an integer code, so the answer is already in the\nfile. Runner recovers that scale and those codes exactly instead of\nre-deriving a scale from the block's extreme value - the derived route is\ncorrect only when a block's codes actually reach zero, and on a block where\nthey do not it saturates the far end of the range and changes values a pure\nrepack had no need to touch. A candidate is accepted only when the value the\ndequantizer will produce equals the source float bit for bit across the whole\nblock, so a source that is not on a grid falls through to the derived scale\nand its output is byte-for-byte what it was before.\n\n`--prune-experts`\n\nrewrites stacked-layout MoE tensors using an explicit JSON\nplan. It is a mechanism, not a quality claim: pruning needs a model-specific\nevaluation against the unpruned parent.\n\n```\n{\"layer_0\":[0,3,7],\"layer_1\":[1,2,5]}\n# Prune only; surviving tensors keep their current quant type.\n./runner -m model.gguf --quantize pruned.gguf --prune-experts keep.json\n\n# Prune and requantize the survivors.\n./runner -m model.gguf --quantize pruned-q4.gguf \\\n  --prune-experts keep.json --quant q4_0\n```\n\nA layer omitted from the plan keeps all experts. Invalid keys, empty lists,\nout-of-range IDs, and unsupported tensor layouts fail instead of silently\nproducing a different model. The layer's router (`ffn_gate_inp`\n\n) and its\nper-expert selection bias (`exp_probs_b`\n\n, in either on-disk spelling - the\n`.weight`\n\nof the DeepSeek-style GGUFs and the `.bias`\n\nof `nemotron_h_moe`\n\n) are\nsliced along with the expert banks, so the survivors in plan order become the\nnew expert index space with no runtime remapping. Non-uniform coverage-pruned\n`nemotron_h_moe`\n\nlayers resolve their expert count from each layer's router and\nrun on CPU; CUDA names and declines this layout because its MoE kernels require\none model-wide expert count.\n`scripts/moe-prune-plan.py`\n\ncan build a plan from calibration data.\n\nArtifacts produced by this project are published only after their stated gate against the named parent. Read each repository's provenance before treating a derivative as equivalent to an original checkpoint.\n\nEvery fidelity claim below is measured under the adopted dual-column bar (margin-qualified top-1 >= 97% AND mean KLD <= 0.05 vs the named parent, 400 teacher-forced positions, zero point exact; plain top-1 always reported beside it).\n\n[Qwen3-30B-A3B selective precision](https://huggingface.co/Joakimpalm-Zen/Qwen3-30B-A3B-selective-attnQ8_0-expQ4_0-GGUF)(attention Q8_0 / experts Q4_0, 17.99 GB)**passes the bar** where the official uniform Q4_K_M fails it, from a byte-verified first-party Q8_0 source. Built with`--type-plan`\n\n; the exact plan is on the card. The artifact class this project now leads with.[Qwen3-Coder-30B keep-120](https://huggingface.co/Joakimpalm-Zen/Qwen3-Coder-30B-A3B-Instruct-keep120-Q4_K_M-GGUF)(expert-pruned, 17.5 GB)**passes both the original and the current bar**- the only published artifact to clear the original bar unaided.[gpt-oss-20b-keep30-MXFP4](https://huggingface.co/Joakimpalm-Zen/gpt-oss-20b-keep30-MXFP4-GGUF)(11.5 GB, 32-to-30-expert derivative)**does not pass the current bar**; its originally published number did not reproduce and the card leads with the measured status. Kept published as a near-miss with its numbers in the open.[gemma-4-E2B-it Q4_K_M/Q4_0 mix](https://huggingface.co/Joakimpalm-Zen/gemma-4-E2B-it-Q4_0-GGUF)(2.63 GB) is the smoke-test artifact from the quickstart: fails the fidelity bar (the card carries the dual-column numbers) and remains the fastest way to try the runner on an 8 GB machine.[NVIDIA-Nemotron-Nano-9B-v2 Q8_0](https://huggingface.co/Joakimpalm-Zen/NVIDIA-Nemotron-Nano-9B-v2-Q8_0-GGUF)(8.81 GB) is not a bar-gated derivative but the first**Mamba-2 hybrid (** artifact the runner supports - a plain, near-lossless Q8_0 of NVIDIA's base, quantised by the runner's own canonical (ggml-byte-identical) quantiser and verified`nemotron_h`\n\n)**5/6 greedy token-identical vs llama.cpp b10353 at Q8_0**(the sole miss a quant-noise near-tie). NVIDIA Open Model License; the card leads with the tool-calling differentiator.- Measurement reports over third-party artifacts, no weights republished,\nevery measured file bound by SHA:\n[Hermes-4-14B quant fidelity](https://huggingface.co/Joakimpalm-Zen/Hermes-4-14B-quant-fidelity-report)(the 4-bit size threshold and the split story), the[Qwen3 speculative pair](https://huggingface.co/Joakimpalm-Zen/Qwen3-speculative-pair-report)(measured draft acceptance, and why the engine's printed tok/round must not be tuned on), and the two Mamba-2 hybrid support reports -[granite-4.0-h-small](https://huggingface.co/Joakimpalm-Zen/granite-4.0-h-small-runner-report)(`granitehybrid`\n\n, 3/5 greedy-identity at the noise floor) and[Nemotron-3.5-Lightning-30B-A3B](https://huggingface.co/Joakimpalm-Zen/Nemotron-3.5-Lightning-30B-A3B-runner-report)(`nemotron_h_moe`\n\n, 4/5) - each carrying its measured-envelope manifest. Two frontier reports (2026-08-20, no weights republished - nothing cleared the bar AND beat upstream): the[Lightning-30B prune frontier](https://huggingface.co/Joakimpalm-Zen/Nemotron-3.5-Lightning-30B-A3B-prune-frontier-report)(keep-126 passes at 99.50%/0.026; the plan is published, the 1.37% saving was not worth an artifact) and the[Muse-Glimmer-30B quant frontier](https://huggingface.co/Joakimpalm-Zen/Muse-Glimmer-30B-runner-quant-frontier-report)(Meta's own Q4_K_M passes the bar; six runner plans measured, none beat it - stated openly).\n\n`runner --help`\n\nremains authoritative for the binary being executed. This\ngrouped reference makes the complete interface discoverable without mixing\nflags into unrelated feature sections.\n\n| Option | Purpose |\n|---|---|\n`-m PATH` |\nGGUF path. In serve mode, `name=path,name2=path2` enables multi-model swap mode. |\n`-p TEXT` |\nOne-shot prompt; escaped sequences such as `\\n` are unescaped. |\n`-f FILE` |\nAppend file contents to the prompt. |\n`-i` |\nStateful interactive chat. |\n`--serve` |\nStart the HTTP server. |\n`--tray` |\nBe the macOS/Windows tray controller instead of running a model. Required where there is no terminal. See\n|\n\n`--no-tray`\n\n`--serve`\n\nand `-i`\n\n.`--port N`\n\n`8080`\n\n.`--parallel N`\n\n`1`\n\n.`--ttl N`\n\n`300`\n\n; `0`\n\ndisables it.`--force-uncertified`\n\n`outside-envelope`\n\nverdict for this runtime (refused by default). See [Measured-envelope gate](#measured-envelope-gate).`--json`\n\n`--json-schema FILE`\n\n`FILE`\n\n.| Option | Purpose |\n|---|---|\n`-n N` |\nMaximum generated tokens, default `256` ; `-1` runs until EOS. |\n`-c N` |\nContext length; default is the smaller of model maximum and 4096. `0` auto-fits with a reservation. |\n`-b N` |\nPrompt batch size, default `64` . Unless `--gpu off` was given, the default is sized from free RAM instead: `512` above 4 GB free, `256` above 1.5 GB, `64` below that, so the tiled prefill GEMM gets more columns per dispatch (measured on Metal/M1: +9% prompt tok/s at 512 over the flat 64 default). `-b` always overrides. |\n`-t N` |\nWorker threads; defaults to physical cores and is capped at `64` . |\n`-s N` |\nRNG seed; default is time-based. `0` is refused: it is the sampler RNG's fixed point, so it cannot produce a stream. |\n`--think` / `--no-think` |\nRequest the model family's thinking or non-thinking prompt shape. With neither flag, Runner renders whatever that family's own reference template renders, which is not the same answer for every family. Families without a distinct thinking prompt accept the flag and ignore it rather than approximate one. |\n`--temp F` |\nTemperature; `0` is greedy and disables repeat penalty. |\n`--top-k N` |\nTop-k sampling; `0` disables it. Several presets ship `0` , where setting it is a measured decode-throughput win that also changes the sampled distribution - see\n`--top-k 40` |\n`--top-p F` |\nNucleus sampling threshold. |\n`--min-p F` |\nProbability floor relative to the top candidate; `0` disables it. |\n`--repeat-penalty F` |\nRecent-token penalty; `1` disables it. |\n`--rope-scale F` |\nForce linear rope position scaling. |\n`--rope-base F` |\nOverride the rope frequency base. |\n`--system TEXT` |\nSystem prompt in interactive chat. |\n`--chat-template NAME` |\nForce `chatml` , `chatml-think` , `llama2` , `llama3` , `mistral` , `mistral-v1` , `mistral-nemo` , `zephyr` , `phi3` , `gemma` , `gemma4` , `gemma4-mainline` , `apertus` , `ornith` , `muse` , `granite` , `harmony` , or `raw` ; default is auto-detection. The three Mistral framings are not interchangeable: `mistral` is the v0.3 / Mistral-Small-2409 form and the fallback for an unrecognised Mistral template, `mistral-v1` is v0.1/v0.2, `mistral-nemo` is Nemo-Instruct-2407. They differ by a space beside each `[INST]` /`[/INST]` marker and by which user turn carries the system prompt - one SentencePiece token per divergent space. gemma-4 likewise ships two chat-template revisions that auto-detect and are byte-exact to their own reference: `gemma4` is the E-series (E2B/E4B) form, `gemma4-mainline` is the 12B/26B-A4B/31B form, which pre-seeds an empty thought block on the thinking-off generation prompt where the E-series pre-seeds nothing. Applies to interactive chat and to `--serve` , including reloads after `/unload` or a `--ttl` expiry. An unrecognized name is an error, and the flag is refused with a swap set (`-m \"name=path,name2=path2\"` ) because it names one template for a set of models that each detect their own - serve that model on its own instance instead. |\n`--no-bos` |\nDo not add the beginning-of-sequence token. |\n`--ignore-eos` |\nContinue generation past end-of-text tokens. |\n\n| Option | Purpose |\n|---|---|\n`--gpu auto|off` |\nAuto-detect offload, or force CPU. |\n`--gpu-layers N` |\nForce the first `N` layers onto the GPU; `0` means no GPU. Omit for auto-fit. On Metal it also overrides the residency veto: a model larger than available RAM is refused for auto-selected partial offload, because nothing pinned can be held resident and the measured result was 8-35x slower decode, but an explicit `--gpu-layers N` splits it anyway. |\n`--cpu-moe [N|auto]` |\nCUDA hybrid placement: keep all, the deepest `N` , or an auto-fit set of expert FFNs in system RAM. |\n`--wait-for-vram [S]` |\nWait for another registered runner to release VRAM, default `300` seconds, instead of failing immediately. |\n`--vram-priority N` |\nAdvisory priority tag on this claim, default `0` (also `RUNNER_VRAM_PRIORITY` ). See\n|\n`--yield-on-request` |\nIn `--serve` , release the resident model at the next idle point when another process has asked it to. See the same section. |\n`--reserve P` |\nLimit this process to `P` percent of total RAM and VRAM. |\n`--reserve-vram P` |\nOverride only the VRAM budget. |\n`--reserve-ram P` |\nOverride only the RAM budget. |\n`--reserve-cpu P` |\nSize the default thread count as a percentage of cores. |\n`--kv f16|q8` |\nKV storage; f16 is default, q8 uses about 53% as much memory and is lossy. |\n`--mlock` |\nAsk the OS to wire mapped weights into RAM; failure is non-fatal. |\n`--moe-prefetch on|off|auto` |\nPrefetch routed expert blocks. Auto enables it only for measured oversubscribed Apple Silicon cases. |\n`--draft PATH` |\nSame-vocabulary draft GGUF for speculative decoding in one-shot, chat, or single-model serve mode. |\n`--draft-k N` |\nDraft tokens per speculative round, default `4` . |\n\n| Option | Purpose |\n|---|---|\n`--quantize OUT` |\nRewrite the loaded model to `OUT` and exit. |\n`--quant q8_0|q4_0|q3_k|q4_k|q6_k|f16|bf16|keep` |\nRequantization target; default `q4_0` , or keep per-tensor types when pruning or merging alone. Requires `--quantize` or `--merge-lora` ; without either the flag is refused rather than ignored. |\n`--type-plan PLAN.json` |\nPer-tensor rewrite plan. First substring rule wins; types are `keep` , `q8_0` , `q4_0` , `q3_k` , `q4_k` , `q6_k` , `f16` , and `bf16` . Example: `{\"default\":\"keep\",\"rules\":[{\"match\":\"_exps.weight\",\"type\":\"q3_k\"}]}` . Requires `--quantize` . |\n`--merge-lora OUT` |\nFold `--lora` into the base weights and write a standalone GGUF that runs in any GGUF runtime: `W' = W + (alpha/r)·B·A` per adapted projection, each tensor requantized to its own type (or `--quant T` ), untouched tensors copied byte-verbatim, `OUT.merge.json` provenance (base/adapter/merged sha256s) written beside it. Deterministic: same inputs, byte-identical merged file. Merging into a quantized type rounds the delta through that type's grid - the merged artifact's fidelity is a measurement, not a given; `base + --lora` remains the exact form. |\n`--prune-experts FILE` |\nApply a per-layer MoE expert keep-list while rewriting. Requires `--quantize` . |\n`--bench-json` |\nRun the built-in prompt/decode benchmark and print JSON metrics. |\n`--lora FILE` , `--lora-scale F` |\nLoad a LoRA adapter GGUF beside the frozen quantized base (llama.cpp adapter naming: `blk.N.<proj>.weight.lora_a/_b` + `adapter.lora.alpha` ; F32, F16 or BF16 tensors - F16 is what llama.cpp's `convert_lora_to_gguf` emits, and a community adapter in that format loads and serves, measured). Interop runs the other way too: an adapter runner trained scores identically (1.000 on its held-out eval) when served by stock llama.cpp. Applied as `y += scale·B(Ax)` on the CPU dense projections (attention q/k/v/output, FFN gate/up/down) - the base weights and kernels are untouched, so every base identity gate still describes the adapted run's substrate. Fails closed by name on shape/rank mismatches, unknown targets, recurrent/gemma-4-MoE architectures, and GPU-resident models (CPU-only for now). A zero adapter is gated byte-identical to the bare base; a real adapter is gated against the merged-weights reference. The adapter id joins the engine's model identity, so cached prefixes never cross an adapter boundary. |\n`--train FILE` , `--train-steps` , `--lr` , `--train-ctx` , `--train-out` , `--save-every` , `--lora-rank` |\nAdamW LoRA training in the serving binary (CPU path, position-batched and threaded under a byte-exact contract - 4B trains at ~20 s/step on a many-core host, 2.3× over the first release, with the adapter bytes gated invariant across binaries, thread counts and the optional `RUNNER_TRAIN_GPU=1` CUDA assist): plain-text corpora or `.jsonl` lines `{\"prompt\",\"completion\",\"weight\"}` with the prompt masked from the loss and per-example weights (the policy-gradient hook `scripts/train-grpo-lite.py` drives). Fresh adapters start as an exact no-op (A seeded, B zero); checkpoints are adapter GGUFs that `--lora` loads back. Deterministic by default: same data + same seed produce a byte-identical adapter file, gated in `make test` . Design, gates and measured results:\n|\n`--score` |\nTeacher-forced scoring: per-token log P(token|prefix) over the raw `-p` /`-f` text - no template, no sampling - printed as JSON (`xyntetik.runner.score.v1` ) with per-position logprobs, NLL and perplexity. The default path scores one forward per position, the exact numerics the sampler sees at decode time; `RUNNER_SCORE_CHUNKED=1` opts into a faster batched pass whose deviation from solo is measured and test-pinned (max |Δlogprob| ~1e-6 on the fixtures - the CPU batched forward is not bit-identical to solo, and scoring defaults to exactness over speed). |\n`--caps` |\nPrint machine, backend, quant, architecture, placement, and sampling capabilities as JSON. |\n`--tool-info` |\nWith `-m` , print the model's tool-call protocol as JSON (`{\"tool_family\":…,\"native_tool_protocol\":…}` ) and exit. No manifest required. |\n`--fit PATH` |\nEstimate whether a GGUF fits this machine and exit. Reads only the header, so a partial download answers the question. |\n`--version` |\nPrint the version and exit. |\n`--parent-pid N` |\nExit when process `N` dies; intended for supervisor cleanup. |\n`-v` |\nPrint verbose model and memory information. |\n\n`--fit`\n\nanswers \"will this run here\" from a model's GGUF **header**, which is\nthe first few megabytes of the file:\n\n``` bash\n$ runner --fit Trinity-Nano-Preview-Q4_K_M.gguf\nfit: Trinity-Nano-Preview-Q4_K_M.gguf\n  model         afmoe, 56 layers, MoE 128 experts, 8 used\n  weights       3.53 GiB\n  hot set       0.66 GiB  (only the routed experts a token actually uses)\n  kv cache      0.22 GiB at ctx 4096, f16   |  0.12 GiB with --kv q8\n  available RAM 3.25 GiB right now\n  verdict       FITS — 2.37 GiB to spare at ctx 4096\n```\n\nThe verdict is `FITS`\n\n, `FITS WITH --kv q8`\n\n, or `PAGES`\n\n, always with the\narithmetic that produced it. `-c N`\n\nsizes the KV estimate for the context you\nactually intend to run. For a sparse MoE the verdict uses the **hot set**, not\nthe file size, because only the routed experts a token selects are touched -\nwhich is why a 3.53 GiB file can be a comfortable fit in 3.25 GiB.\n\nThe runner does not download anything, and `--fit`\n\nis not a reason to teach it\nHTTP. Fetch a header yourself with a ranged read - 16 MiB covers a large\nvocabulary; smaller models need far less:\n\n```\ncurl -r 0-16777215 -L -o head.gguf \\\n  https://huggingface.co/ORG/REPO/resolve/main/MODEL.gguf\nrunner --fit head.gguf\n```\n\nThe sizes reported from a truncated header are the **whole** model's, because\nthey come from the tensor descriptors rather than from how many bytes arrived.\nLoading such a file still fails, as it should: the normal loader refuses a GGUF\nwhose data section does not cover the tensors it declares, and `--fit`\n\nreads\nthrough a separate path rather than relaxing that check.\n\nUse chat mode or an API chat surface to judge an instruction-tuned model.\nRaw `-p`\n\ncompletion deliberately bypasses chat framing and is primarily useful\nfor benchmarks and deterministic comparison gates.\n\nSampling defaults come from a per-family preset selected from model metadata\nand filename. The chosen preset is logged at load, `--caps`\n\npublishes the full\npreset table, and explicit sampling flags always win. At `--temp 0`\n\n, runner\nreturns the model argmax without applying repeat penalty.\n\nInteractive chat keeps its KV state across turns and auto-detects the template from metadata and vocabulary. Thinking channels are displayed separately. The server additionally reuses the longest shared prompt prefix across requests.\n\nOn macOS and Windows, a session you sit with - a bare invocation, `--serve`\n\n, or\n`-i`\n\n- also raises the desktop tray, which is left running afterwards. One-shot\n`-p`\n\nruns, tooling modes, pipes, scripts, CI, and Linux keep text-mode\nbehavior, and `--no-tray`\n\nopts out everywhere. See [Desktop tray](#desktop-tray).\n\nCPU execution has portable scalar kernels plus AVX2/FMA and ARM NEON paths.\n`--gpu auto`\n\nselects a usable backend and falls back with a reason when a model\nlayout, tensor type, runtime, or capacity is unsupported.\n\n| Backend | Tensor formats |\n|---|---|\n| CPU | F32, F16, BF16, Q8_0, Q4_0, Q4_1, Q5_0, Q5_1, Q2_K, Q3_K, Q4_K, Q5_K, Q6_K, IQ4_NL, IQ4_XS, MXFP4 |\n| Metal | The full CPU list |\n| CUDA | The full CPU list |\n\n`runner --caps`\n\nis the live source of truth for a particular executable and\nmachine. Architecture and MoE layout checks still happen at model load; a\nlisted tensor kernel does not imply that every architecture using that tensor\nis implemented on that backend.\n\n**Metal:** Apple Silicon uses zero-copy mapped weights and unified-memory KV.\nMetal supports f16 and q8 KV, dense and selected MoE layouts, and tiled prefill\nGEMMs. Full offload is the preferred and default shape. A file above\n`gpu.max_working_set_bytes`\n\nin `--caps`\n\ntakes a leading-layer split when the\ntensor layout allows a contiguous prefix wrap *and* the whole model still fits\nin RAM; when it does not, the backend falls back to CPU rather than split,\nbecause pinning part of a model that does not fit measured 8–35x slower than\nCPU-only on an 8 GB M1. `--gpu-layers N`\n\nforces a split anyway. Multi-part (split)\nGGUF sets also run on the CPU: the zero-copy wrap addresses one mapping and a\nsplit set has one per part, so the backend declines the model at load rather\nthan binding weights it cannot place. The embedded shader\ngate compiles the library and verifies every kernel the backend looks up,\nreading that roster out of `src/metal.m`\n\nrather than restating it.\n\n**CUDA:** Linux and Windows use the dynamically loaded driver API and embedded\n`sm_75`\n\nPTX. **The embedded PTX is built by the CUDA 13.0 toolchain (PTX ISA\n9.0), so GPU execution requires a driver with CUDA 13.0 support or newer (the\nR580 series)** - see the driver note in the Install section; on an older\ndriver the runner reports the JIT failure and falls back to CPU. Full and\npartial layer offload are supported. Sparse MoE can keep expert FFNs in RAM\nwith `--cpu-moe`\n\nwhile attention and dense tensors remain on the GPU.\n`make ptx`\n\nregenerates the embedded header and requires a CUDA toolkit only\nfor that development step.\n\nScalar-path CPU/GPU identity is an evidence result, not a property inferred from a backend name. CUDA tensor-core and Metal tiled prefill kernels reassociate floating-point sums, so they are promoted by teacher-forced tolerance tests. CUDA currently promotes Q4_K/Q6_K/Q8_0 on the gated dense families and Q4_0 on Gemma 4; the latter was bit-identical over 820 tensor-core dispatches on the real 31B QAT artifact.\n\nOn Metal that now covers **decode as well as prefill**: the cooperative KV\nattention read was promoted on 2026-08-17 after clearing zero teacher-forced\ntop-1 flips out of 64 on every local model that reaches it - gemma-4 E2B,\ngemma-3-4B, granite-4.1-8B under a layer split, SmolLM2, and the NoPE /\nattention-temperature fixtures - in both f16 and q8 KV cache formats, for a\nmeasured +3.0–4.3 % decode across 2.3k–8.1k token spans. So Metal decode at\nlong context is a tolerance-gated route, not a byte-identical one.\n\n`RUNNER_CUDA_TC=0`\n\n, `RUNNER_METAL_MM=0`\n\nand `RUNNER_METAL_ATTN_COOP=0`\n\npin the\nbyte-identical scalar paths for identity investigations; every CPU-vs-GPU byte\ncomparison in the test suite sets them. `./test-attn-tol MODEL.gguf`\n\nis the\nattention gate.\nWeights are wrapped zero-copy from the model mmap. A file larger than the\ndevice's `maxBufferLength`\n\n- 4.29 GB on an M1, against a 5.73 GB working set -\nis wrapped in several buffers instead of being copied or forced into a\nCPU/GPU layer split. The cuts fall on tensor boundaries, so no tensor spans two\nbuffers and output is byte-identical to a single-buffer wrap;\n`RUNNER_METAL_MAX_BUF`\n\nshrinks the per-buffer ceiling so that path can be\nexercised on a machine whose models all fit one buffer, and\n`make test-metal-multibuf`\n\nis the byte-identity gate. A separate pure admission\ngate simulates a file above `maxBufferLength`\n\nbut below the aggregate working\nset, ensuring it remains a full offload. A single tensor larger than the\nper-buffer ceiling still cannot be wrapped and says so.\n\n`RUNNER_METAL_ATTN_COOP=0`\n\npins the byte-identical decode attention kernel.\nThe default is the cooperative KV read: one simdgroup owns a KV row and its\nlanes split `head_dim`\n\n, so a load covers 32 consecutive elements instead of 32\nrows. It reassociates the per-row dot into a `simd_sum`\n\n, which is why it\nanswers to `./test-attn-tol`\n\nrather than to an identity claim.\n\n`RUNNER_METAL_MV=1`\n\nopts into a reassociating Metal *decode* matvec (q4_0/q8_0,\nfloat4 accumulation and the q4_0 zero-point factored out of the inner loop).\nIt clears the 0/64 teacher-forced flip bar on both formats but measured\nneutral on an 8-core M1 - −0.16 % bandwidth-bound, −0.01 % dispatch-bound - so\nit is **off by default**, leaving the byte-identical kernel on the default\npath. `./test-mv-tol MODEL.gguf`\n\nis the gate; see\n`docs/negative-result-metal-multirow-matvec.md`\n\nfor why decode on that machine\nis bound by bytes rather than instructions.\nThe CPU quant dot/dequant module is a separate translation unit compiled with\n`-fno-fast-math`\n\n; fast math remains enabled for the rest of the engine.\n\n**CPU:** the x86 dot kernels read weights in their on-disk quantized form and\nkeep f32 activations, which is token-identical across builds and thread\ncounts. `RUNNER_CPU_I8=1`\n\nopts into a fused int8 decode dot (AVX-512 VNNI,\nAVX2 fallback): 2.4-2.5x on the kernel in isolation, but it quantizes the\nactivations, so it is **off by default** - no format cleared the 0/64\nteacher-forced flip bar with a decode gain worth taking on the measurement\nbox. `./test-i8-tol MODEL.gguf`\n\nis the gate.\n`RUNNER_TPOOL_SPIN`\n\nsets how many relax iterations a pool worker spins before\nparking (default 3000, roughly 50 us); `0`\n\nrestores a pure condvar pool. The\nspin window only changes when threads wake, never which rows they compute, so\noutput is unaffected either way. See [docs/performance.md](/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/performance.md).\n\nVulkan is not implemented; AMD and Intel GPUs use the CPU path.\n\n- A requested context above the training length applies model metadata for\nlinear/YaRN/llama-3 rope scaling, or automatic YaRN extension when metadata\ndoes not supply a native scheme.\n`--rope-scale`\n\nand`--rope-base`\n\noverride that behavior. `--kv q8`\n\nstores q8_0 blocks when every layer's head dimension is divisible by 32. It works on CPU, CUDA, and Metal, participates in capacity auto-fit, and is intentionally not token-identical to f16 KV. An incompatible head dimension is reported at load and keeps the cache in f16.- Prompt evaluation is batched;\n`-b`\n\ncontrols the batch and`-v`\n\nprints the KV allocation before inference.\n\n`--reserve`\n\nand its RAM, VRAM, and CPU variants let runner coexist with other\nworkloads. With `-c 0`\n\n, the context grows into the remaining reservation up to\nthe model's training context. A cross-process registry prevents a second\nrunner from blindly consuming occupied VRAM; `--wait-for-vram`\n\nturns that\nrefusal into a bounded queue.\n\n`--mlock`\n\ncan prevent mapped weights from being evicted, but should not be used\nto force a model larger than available RAM to stay resident. Sparse MoE load\nlogs distinguish total file size from the smaller per-token hot set.\n\nOn high-core-count hosts, sparse MoE decode can be memory-bandwidth bound well\nbefore the 64-thread cap. Measure `-t 12`\n\nto `-t 16`\n\nas well as the default;\nthe project recorded 17.0 tok/s at 12-16 threads versus 7.8 tok/s at 64 on one\n128-core gemma-4-26B-A4B run. This is workload evidence, not a universal\nthread-count rule.\n\nThe VRAM registry (above) accounts for who holds what; these three primitives\nlet cooperating processes negotiate around that accounting without turning\nrunner into a scheduler. All of it is **advisory**: it only has any effect on\nprocesses that opt in by passing the flags below, and nothing in the engine\ncan force, signal, or kill an uncooperative one. Fair-share, priority lanes,\nstarvation prevention, and actual preemption are policy, and policy lives in\nwhatever coordinates several runner instances, not in the engine - this is the\nraw material for that layer, not the layer itself.\n\n**Priority tag.**`--vram-priority N`\n\n(default`0`\n\n, also`RUNNER_VRAM_PRIORITY`\n\n) records a small-integer tag on the claim. It is printed in the refusal listing next to pid, model, bytes, and uptime -`pid 4821 holding 5.2GB for Qwen3-4B-Q4_K_M, up 4h39m, priority 3`\n\n. A ledger entry written by a runner built before this field has exactly 7 tab-separated columns instead of 8 and is read as priority`0`\n\n, the same as an explicit`--vram-priority 0`\n\n.**Priority-ordered waiting.** Among several`--wait-for-vram`\n\nwaiters queued on the same GPU, a higher-priority one is admitted first once space frees - but only among waiters whose own request currently fits that freed space; a high-priority ask that does not fit yet never blocks a smaller low-priority one out of room it does not need. This is ordering among cooperating waiters, not a reservation: a process that never passes`--wait-for-vram`\n\n, or that claims VRAM some other way, is invisible to it and can still take memory out of turn.**Cooperative yield.**`--serve --yield-on-request`\n\nopts a resident model into releasing itself when asked. The ask is a REQUEST, checked only at the one place`--serve`\n\nis ever idle between requests - never mid-generation, never by a signal. An opted-in holder that sees one logs why and unloads cleanly, the same path`--ttl`\n\nand`POST /unload`\n\nalready use. An unopted-in holder, or one that is busy, never notices. Nothing here is preemption: there is no timeout after which a holder is forced out.\n\nNone of the three needs a GPU to exercise - `tests/test_vram_registry.c`\n\ndrives the whole surface, including priority ordering, through the same\nsynthetic free-VRAM callback the rest of the registry's tests use.\n\nRunner already refuses to treat output as correct without a schema contract.\nThe measured-envelope gate extends that one layer down, to the model itself. A\ncertification run records what was actually *measured* for one artifact on one\nruntime - the CPU==GPU identity check, the fidelity gate, whether the model\nfits its memory class - into a `<model>.gguf.envelope.json`\n\nsidecar. At load\nRunner reads the sidecar sitting next to the model and resolves it against the\nruntime it is actually running (`runner --version`\n\nand the model's active\ncompute backend,\nexact-match - a manifest measured on a different version or backend does not\nspeak for this one):\n\nThe states are distinguished by what Runner actually *knows* about the model,\nnot just by what they do - two of them load with a banner but mean different\nthings:\n\n| State | Condition | Behavior |\n|---|---|---|\ncertified |\nthe sidecar matches this runtime and its gate passed | loads; a banner notes the match |\noutside-envelope |\nthe sidecar matches this runtime and records a measured refusal (e.g. the model does not fit, or an identity check failed) | refused at load with the measured reason; `--force-uncertified` overrides with a loud warning |\nexperimental |\nthe sidecar matches this runtime and its verdict is literally `experimental` - a real measurement that came back inconclusive |\nloads; a banner notes it is not certified |\nindeterminate |\na sidecar is present but cannot be used to judge this run - unreadable, an unknown schema, or measured on a different runtime/backend |\nloads (fail-open); a banner notes it could not be judged |\nunclassified |\nno sidecar at all | loads silently - a transitional/legacy state |\n\nTwo of those distinctions are load-bearing. **unclassified** ≠ *experimental*: a\nmodel with no sidecar predates or sits outside the certification pipeline, so\nthere is nothing measured to report - not a measurement that came back\ninconclusive - and it does not warrant a banner on every load. **indeterminate**\n≠ *experimental* either: \"we could not read/apply the sidecar\" is not the same\nclaim as \"we measured this and it was inconclusive.\" As the pipeline's coverage\ngrows, unclassified is the state that shrinks.\n\nThe rest of the wording is deliberate too: a configuration *matches a measured\nenvelope*, it is not *certified* as a standing property - the claim is scoped to\nthat exact artifact, backend, and date. The gate is fail-open on doubt: only a\n*matching* `outside-envelope`\n\nverdict ever refuses; anything unreadable, foreign,\nor unrecognized is indeterminate and loads, because a wrong refusal is worse than\nnone. Runner only ever *reads* this file; it is produced by the certification\npipeline, never at runtime.\n\nA manifest may also carry an optional `tool_calling`\n\nblock: a summary of how the\nmodel behaves under tool use - engine truncation-recovery, whether the tool-call\n*schema shape* still holds at a low quant, an agent-torture pass/fail, and the\nmodel's native tool protocol. This axis is **reported-only**: it changes no\nverdict and never refuses a load. When the block is present, Runner prints one\nextra banner line at load, showing only the sub-fields that were actually\nmeasured, for example:\n\n```\nenvelope: tool-calling gate=pass — truncation 6/6, schema-shape@Q4_0, agent-torture pass, native granite\n```\n\nA manifest with no `tool_calling`\n\nblock prints nothing extra. You can also query\na model's native tool protocol directly, without any manifest, with `runner --tool-info -m model.gguf`\n\n, which prints\n`{\"tool_family\":…,\"native_tool_protocol\":…}`\n\n. The full block, the evidence each\nfield comes from, and the honesty caveats (notably that *schema-shape holding at\nQ4_0* is about the call\n\n**shape**, not the argument values) are documented in\n\n[docs/envelope-manifests/README.md](/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/envelope-manifests/README.md).\n\nStart a single-model server:\n\n```\n./runner -m model.gguf --serve --port 8080 --parallel 2\n```\n\nThe server is HTTP on loopback only, with no TLS or authentication. Binding to\n`127.0.0.1`\n\nis an invariant rather than a default: there is no host flag,\nenvironment variable, config key, or local-network toggle that can expose it.\nPut it behind an authenticated reverse proxy or tunnel when remote access is\nneeded; do not forward the port directly. Host and Origin validation rejects\nnon-loopback authorities.\n\n| Method and path | Purpose |\n|---|---|\n`POST /v1/chat/completions` |\nOpenAI Chat Completions, including SSE, tools, structured output, logprobs, and stop strings. |\n`POST /v1/responses` |\nOpenAI Responses translation over the same engine and tool envelope. |\n`POST /v1/completions` |\nLegacy raw prompt completions. |\n`POST /v1/embeddings` |\nMean-pooled, L2-normalized embeddings. |\n`POST /v1/messages` |\nAnthropic Messages translation. |\n`POST /v1/messages/count_tokens` |\nToken count for the matching Messages request. |\n`GET /v1/models` |\nRegistered models and current residency. |\n`GET /v1/capabilities` |\nActive model, sampling preset, and optional Xyntetik agent profile. |\n`GET /v1/runner/prefix-cache` |\nPrefix-cache size, limits, and counters. |\n`POST /v1/runner/prefix-cache/clear` |\nRelease cached prefixes without unloading the model. |\n`GET /health` |\nServer and resident-model health, plus this process's `rss_bytes` /`peak_rss_bytes` and cumulative `tokens_prompt` , `tokens_generated` , `generate_seconds` , `batch_steps` and `batch_sequences` . |\n`POST /unload` |\nRelease resident model, draft and prefix-cache memory; the next request reloads on demand. Deferred to the next safe point while a load or generation is in flight (the reply says `\"deferred\":true` ). Needs the registry: a server without one refuses with `409` rather than reporting a success it cannot deliver - see the residency note below. |\n\n`GET /unload`\n\nis deliberately refused with `405`\n\n; unloading is a state change.\n\nBuffered generation responses include `runner_telemetry`\n\nwith prompt tokens\nreused/evaluated, generation timing, paging counters, and structured or\nspeculative mode flags. `speculative`\n\nreports whether that request used the\nspeculative walk, not merely whether the server has a draft loaded; logprob and\nchoice-logprob capture use the solo walk and therefore report it as false. Set\nrequest field `\"cache_prompt\": false`\n\nto bypass prefix reuse. Streaming clients\nwhose writes fail cancel generation. An orderly client socket close on any\ncompletion surface also cancels at the next complete prefill chunk or decode\nstep, so an abandoned long prompt does not keep its slot busy; the probe is\nnon-consuming, so an alive quiet client or readable pipelined bytes are not a\ncancellation signal.\n\nEvery generating endpoint also accepts a per-request `\"timeout\"`\n\nin seconds\n(`0`\n\n–`86400`\n\n), which overrides `RUNNER_REQUEST_TIMEOUT`\n\nfor that request; `0`\n\nmeans no limit and an out-of-range value is a `400`\n\n. Expiry is a truncation,\nnot an error: generation ends, `finish_reason`\n\nis `\"length\"`\n\n, and constrained\noutput is closed to a legal document exactly as a token-ceiling hit would be.\n\nPrefix reuse lives in this process only. The cache is host RAM bounded by\n`RUNNER_PREFIX_CACHE_MB`\n\n, and it is released by `POST /unload`\n\n, by\n`POST /v1/runner/prefix-cache/clear`\n\n, by a `keep_alive: 0`\n\nrequest, and at\nexit. A model swap deliberately keeps it - surviving a swap is the point of\nsnapshotting a prefix rather than holding a slot - and every entry is bound to\nthe model, geometry, tokenizer, context length and KV element type it was\ntaken from, so another model cannot install one. There is no on-disk warm\nstart: a restarted server prefills from cold.\n\n`--parallel N`\n\ncreates independent KV caches and thread pools while sharing\nmapped weights. Threads are divided across slots. Multi-model swap mode uses\none slot because only one model is resident at a time, and accepts up to 16\nregistered models:\n\n```\n./runner -m \"code=qwen3-14b.gguf,fast=qwen3-4b.gguf\" \\\n  --serve --ttl 300\n```\n\nEach request selects the registered name in its `model`\n\nfield.\n\nResidency control - `--ttl`\n\n, `POST /unload`\n\n, and the per-request `keep_alive`\n\n(seconds; `0`\n\nunloads at the next safe point, negative pins the model) - needs\nthe model registry, which is not the same line as \"swap mode\": a single model\nserved with the default `--parallel 1`\n\njoins the registry as a one-entry set,\nso all three work there exactly as they do for a swap set. The exception is a\nmulti-slot single-model server (`--parallel N`\n\nwith `N > 1`\n\n): its slots hold\nthe model directly, with no registry to unload it from. There `POST /unload`\n\n**refuses** with `409`\n\nand an error naming the configuration - it used to\nanswer `{\"status\":\"ok\"}`\n\nafter freeing only the prefix cache, which told an\noperator reclaiming memory that weights and KV were gone while every byte\nstayed resident. A completion that carries a `keep_alive`\n\nfield there is\n**refused** with `400`\n\nfor the same reason: the field is well-formed but not\nsatisfiable without a registry, and `keep_alive: 0`\n\nwould free nothing - it\nused to be range-checked and then silently dropped. A completion with no\n`keep_alive`\n\nfield is the normal case and is unaffected.\n`POST /v1/runner/prefix-cache/clear`\n\nworks everywhere and is what both\nrefusals point at; serve with `--parallel 1`\n\nif you need an unloadable\nserver.\n\nThese environment variables are operator controls rather than hidden feature switches:\n\n| Variable | Default | Purpose |\n|---|---|---|\n`RUNNER_MAX_QUEUE` |\n`512` |\nLower the fixed admission queue capacity. |\n`RUNNER_REQUEST_TIMEOUT` |\n`0` |\nDefault generation wall-clock limit in seconds; `0` disables it. |\n`RUNNER_PREFIX_CACHE_MB` |\n`512` |\nHost-RAM budget for shared prompt prefixes; `0` disables storage. |\n`RUNNER_PREFIX_CACHE_TTL` |\n`600` |\nPrefix idle lifetime in seconds. |\n`RUNNER_MOE_PREFETCH` |\nper-machine auto | Compatibility fallback for `--moe-prefetch` ; the CLI flag has precedence. `0` /`off` disables it and other non-empty values enable it. |\n`RUNNER_ALLOW_UNKNOWN_ARCH` |\nunset | Admit a GGUF whose `general.architecture` this binary does not implement, running it through llama-style math. Unset, such a file is refused at load. Set, the load is attempted and a warning says the output may be silently wrong. Experimental, not a supported configuration. |\n`RUNNER_VRAM_PRIORITY` |\n`0` |\nBaseline for `--vram-priority` ; the flag overrides it. |\n\nBeyond these, the binary reads a number of development switches -\n`RUNNER_DEBUG_TOKENS`\n\n, `RUNNER_DEBUG_ACT`\n\n, `RUNNER_MOE_TRACE`\n\n,\n`RUNNER_LAYER_SIM`\n\n, `RUNNER_GRAMMAR_TRACE`\n\n, `RUNNER_SCHEMA_TRACE`\n\n, the\n`RUNNER_METAL_*`\n\n/`RUNNER_CUDA_*`\n\nkernel knobs and failure injectors. They print\nor dump internals for the tools under `scripts/`\n\n(`moe-prune-plan.py`\n\nconsumes\n`RUNNER_MOE_TRACE`\n\n, `classify-grammar-trace.py`\n\nconsumes\n`RUNNER_GRAMMAR_TRACE`\n\n) and are read at first use. They are instrumentation,\nnot interface: names, formats and defaults change without notice, and nothing\noutside this repository should depend on them.\n\nGGUF exports may opt into the versioned `gridcore.agent.*`\n\nprofile. Runner\nvalidates its protocol/tokenizer versions, schema identity, digest, and\nrequired runtime features before allocating model state; unknown requirements\nfail closed. `GET /v1/capabilities`\n\nreturns the admitted profile. See\n[docs/agent-profile.md](/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/agent-profile.md).\n\nChat supports buffered and SSE responses, part-array content, assistant\n`tool_calls`\n\nhistory, `role:\"tool\"`\n\nresults, `stream_options.include_usage`\n\n,\n`logprobs`\n\n/`top_logprobs`\n\n, `min_p`\n\n, `repeat_penalty`\n\n, up to four stop strings,\nand `keep_alive`\n\non a registry-backed server. Tool declarations are rendered into the model\nprompt in the resident model's native tool protocol - identically on\n`/v1/chat/completions`\n\n, `/v1/responses`\n\n, and `/v1/messages`\n\n- and constrained\nback into well-formed `tool_calls`\n\n.\n\nStop strings and tool declarations cannot be combined: a request carrying both\n`stop`\n\n(or Anthropic's `stop_sequences`\n\n) and `tools`\n\nis refused with HTTP 400.\nA stop string is a rule about the model's visible text, but under the tool\nenvelope the model generates protocol - Harmony channel markers and recipient\nheaders, Muse's `<atem:invoke>`\n\nblocks, gemma-4's `<|tool_call>`\n\nblocks, or the\ngeneric envelope's own JSON syntax - and the caller receives\nonly the demultiplexed result. Matching stop strings against that document\nfires on framing nobody wrote (`[\"\\n\\n\"]`\n\n, `[\"}\"]`\n\nand `[\"<|\"]`\n\nall hit).\nRunner refuses the request rather than ignoring the field. The refusal is a\nsemantic one: a stop match no longer corrupts the document - under a plain\n`response_format`\n\nit truncates the constraint validator with it, described\nunder structured output below - but a rule the caller wrote about visible text\ncannot be honoured against protocol the caller never sees.\n\n`parallel_tool_calls:true`\n\ncompiles the generic JSON tool envelope into a\nbounded `{\"calls\":[...]}`\n\narray (up to 8 entries) over the same discriminated\nunion, instead of a single object; a direct answer is just a one-element\narray holding the `final`\n\nbranch. Buffered and streaming requests map it the\nsame way: each call gets its own `tool_calls[].index`\n\n, announced and closed\nbefore the next one opens, so a client reassembles a parallel SSE turn with\nthe identical per-index accumulation it already uses for one call. A budget\nthat truncates mid-call still closes to a legal, executable document -\n`sval_close`\n\nguarantees that - but `finish_reason`\n\nstays `\"length\"`\n\n, never\n`\"tool_calls\"`\n\n, when the closer rather than the model finished the entry.\n\ngpt-oss uses its trained Harmony tool protocol instead of that generic\nenvelope. Runner renders the official TypeScript `# Tools`\n\nnamespace in the\nHarmony **developer** turn - the slot the reference reserves for OpenAI\nfunction tools, after `# Instructions`\n\nand separated from it by a blank line,\nor alone in a developer turn of its own when the caller sent no system\nmessage. The system turn has a second `# Tools`\n\nslot that renders identical\nbytes, but it is for the model's built-in browser/python tools and Runner\nnever uses it. Declaring tools also appends `Calls to these tools must go to the commentary channel: 'functions'.`\n\nto the system turn, on the line after\n`# Valid channels`\n\n; that channel list is the constant `analysis, commentary, final`\n\nwhether or not tools are declared. Runner then constrains the\ngenerated recipient to `functions.NAME`\n\n,\nconstrains the JSON after `<|constrain|>`\n\nagainst that function's declared\nparameters, and maps the native `commentary`\n\n/`<|call|>`\n\nturn back to ordinary\nOpenAI `tool_calls`\n\n. `tool_choice`\n\n(`auto`\n\n, `required`\n\n, named, and `none`\n\n),\nJSON-schema `response_format`\n\non the auto/final branch, buffered replies, SSE,\nreasoning, visible commentary before a call, tool-result replay, and histories\ncontaining several prior calls all use the same native path. Harmony ends one\nsampled turn at its first `<|call|>`\n\n; therefore `parallel_tool_calls:true`\n\npermits a call but does not fabricate several calls inside one Harmony turn.\nMultiple calls are replayed as consecutive native turns when the client sends\nthem in history. A replayed tool result is spelled\n`<|start|>functions.NAME to=assistant<|channel|>commentary`\n\n: the recipient is\nnot decoration, because the reference resolves the author token before the\nchannel and accepts a namespaced author as the tool role only through that\n`to=`\n\nbranch. It must also be attributable, because the turn is authored by the\nfunction that ran: runner resolves that name from the call the result answers -\n`tool_call_id`\n\non Chat, `call_id`\n\non Responses, `tool_use_id`\n\non Messages - and\nfalls back to the sole declared function when exactly one tool is declared,\nsince there is no other function in the namespace the result could be from.\nWhen the lookup finds nothing and two or more tools are declared, the request is\nrefused with a 400 naming the field that would fix it, rather than rendering a\nturn shape gpt-oss was never trained on or a function name invented from an\nidentifier. Runner is stateless, so a client that keeps its own history has to\nsend the call item back alongside its result. The `# Tools`\n\nTypeScript follows\nthe openai-harmony reference renderer rather than TypeScript validity, so that\none tool schema yields one prompt across engines instead of a per-engine\nspelling. Only the tool-level description is split into one `// `\n\ncomment per\nline; an object schema's own description, a property title, and a property\ndescription each take a single `// `\n\nprefix, which leaves a multi-line value's\ncontinuation as a bare uncommented line. That is the reference's own quirk,\nreproduced deliberately and pinned by goldens rendered through openai-harmony\n0.0.8 (abd677f7) via `DeveloperContent.with_function_tools`\n\n- the\nfunction-tool slot, named in each golden's comment because the builtin-tool\nslot renders the same bytes in the wrong turn - and cross-checked against the\n`chat_template`\n\nembedded in the official gpt-oss GGUF. Strict Harmony tool turns bound a pre-call analysis or visible\ncommentary message to 192 UTF-8 bytes; at that boundary the trained assistant\nhandoff is forced, preserving enough output budget for model-generated\narguments instead of letting a turn narrate its intent forever. The bound is\nwhat ends that narration whenever a call is legal - not only under\n`tool_choice:\"required\"`\n\n- and it is not free: a tool result the model wants to\nquote back, typically a JSON document, can be cut mid-quotation, and gpt-oss\nthen repeats the call it was just answered before replying on the next round. A\nprose tool result fits inside the bound and answers directly. Lifting the bound\nis measured and worse, not untried:\n[docs/negative-result-harmony-analysis-bound.md](/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/negative-result-harmony-analysis-bound.md).\n\nFor example, the usual OpenAI request needs no Runner-specific switch:\n\n```\n{\n  \"model\": \"gpt-oss-20b\",\n  \"messages\": [{\"role\": \"user\", \"content\": \"What is the weather in Oslo?\"}],\n  \"tools\": [{\"type\": \"function\", \"function\": {\n    \"name\": \"get_weather\",\n    \"description\": \"Get the current weather\",\n    \"parameters\": {\"type\": \"object\", \"properties\": {\n      \"city\": {\"type\": \"string\"}\n    }, \"required\": [\"city\"]}\n  }}],\n  \"tool_choice\": \"required\"\n}\n```\n\nMuse's native atem format carries scalar parameter values as raw text rather\nthan JSON strings. Consequently a scalar value cannot contain the literal\n`</atem:parameter>`\n\nsequence: atem itself uses that sentinel as the value\nboundary and its reference template describes the output as regex-parsed,\nnot XML-escaped. Declared parameters retain their schema optionality: members\nlisted in `required`\n\nare forced, while other members may be omitted in their\ndeclared order.\n\nFor Muse, the recipient header is part of the constrained turn: `to=user`\n\nselects a plain answer and a declared tool recipient pins the matching\n`<atem:invoke>`\n\nname. Buffered and SSE parsing collect consecutive native\ncalls separated by `<|eom|>`\n\ninto ordered OpenAI `tool_calls`\n\n; the separator\nis not treated as a global stop token.\n\nNative atem calling is selected automatically when a loaded Muse Glimmer\nmodel receives `tools`\n\n. Set `atem_tool_calling:false`\n\non a Chat Completions\nrequest to use Runner's generic JSON-schema tool envelope instead; its payload\nis still constrained behind Muse's `to=user`\n\nrecipient header, so the override\ndoes not leak prompt syntax into `content`\n\n. `tool_choice`\n\n(`auto`\n\n, `required`\n\n,\nnamed, and `none`\n\n) still controls the allowed recipients.\n`parallel_tool_calls:true`\n\nwith a required/named native choice constrains a\nbounded two-call turn. With native `tool_choice:\"auto\"`\n\n, the same flag retains\nthe auto turn and therefore permits at most one call. Families without a native\nconstrained-generation protocol keep the generic JSON-schema output path.\ngpt-oss, Muse Glimmer and gemma-4 have native constrained-generation paths;\nApertus uses its reference template's native declarations and history framing,\nwhile its generated output remains on the generic strict envelope.\n\nWhen native `tool_choice:\"auto\"`\n\nis combined with a JSON-schema\n`response_format`\n\n, the `to=user`\n\nalternative is compiled against that final\nschema; choosing not to call a tool therefore does not weaken structured\noutput.\n\nMuse's own protocol tokens satisfy the constraint's spelled markers: when the\nmodel emits a control token such as `<|message|>`\n\nwhere the automaton expects\nthat literal, the engine advances the automaton and the output stream with its\nspelling, so constrained generation follows the model's trained header format\ninstead of forcing it to type protocol out as text.\n\nAn explicit `enable_thinking:true`\n\nstarts Muse's self-addressed reasoning turn\nbefore the recipient constraint. If generation is cut at the token limit,\nthe atem automaton closes the current parameter/invoke/function-call tail;\nraw scalar recovery uses the declared parameter type so the resulting OpenAI\narguments document remains executable, and an incomplete string enum is\ncompleted to the member sharing its longest prefix. Missing numeric text is\nrecovered within the parameter's declared bounds. A native `to=user`\n\ntext answer ends at\nthe model's own end-of-turn token and reports `finish_reason:\"stop\"`\n\n; only a\ngenuine token-limit cut reports `\"length\"`\n\n.\n\nA model that opens a thinking block is bounded differently depending on whether the request asked for a structured deliverable, and the asymmetry is deliberate:\n\n**With a constraint**-`response_format`\n\n(`json_object`\n\n/`json_schema`\n\n) or`tools`\n\n- the thinking prelude is capped at half the token budget. Hitting that cap does not end the turn: the prelude is closed and the remaining budget goes on the payload that was actually requested.`finish_reason`\n\nis the standard`\"length\"`\n\n, and`runner_telemetry.finish_detail`\n\ncarries`\"reasoning_limit\"`\n\nso the specific cause stays recoverable. This is the shape Anthropic's extended thinking uses - thinking has its own budget under`max_tokens`\n\n, and the answer is still produced.**Without one**, there is no prelude cap. The turn runs to`max_tokens`\n\nlike any other and`finish_reason`\n\nis a plain`\"length\"`\n\nwith no`finish_detail`\n\n. This resembles OpenAI's reasoning models, where reasoning and output share one ceiling and a reasoning-heavy turn can return little or nothing.\n\nThe reason for the split: under a constraint the caller is owed a document, and\nreturning an empty one is a worse answer than a shorter thought. Measured on\ngemma-4-E2B, two of four tool prompts opened a thinking block and never closed\nit - with `-n 200`\n\nthat burned 100 tokens and returned a single newline.\n\nA loaded gemma-4 model that receives `tools`\n\ndeclares them the way its own\nchat template does - `<|tool>declaration:NAME{description:<|\"|>...<|\"|>, parameters:{...}}<tool|>`\n\n, inside the caller's system turn rather than in a\nprepended one - and calls them as `<|tool_call>call:NAME{city:<|\"|>Oslo<|\"|>} <tool_call|>`\n\n, with results replayed as `<|tool_response>response:NAME{...} <tool_response|>`\n\nfrom inside the model turn that made the call. Those bytes\nare compared against the reference template case by case in\n`scripts/template-conformance.py`\n\n.\n\nThe native syntax does not cost the strict envelope. The generated turn is\nstill constrained: the tool name comes from an enumeration of the declared\nfunctions, each argument key and type from that function's schema, and\n`tool_choice`\n\n(`auto`\n\n, `required`\n\n, named, `none`\n\n) selects which branches\nexist at all - `required`\n\nremoves the prose branch, which is what enforcement\nmeans here. A call cut off by the token limit is closed to the smallest legal\nending and still reports `finish_reason:\"length\"`\n\n. What the client receives is\nordinary JSON: `arguments`\n\nis translated out of gemma4's `<|\"|>`\n\nspelling on\nboth the buffered and the streamed path, so no native framing reaches an\nOpenAI client.\n\nDeclared parameters retain their schema optionality: members listed in\n`required`\n\nare forced, while other members may be omitted without changing\ngemma4's dict-sorted native order. One limit is worth knowing before you write\na schema for this family: a parameter with no declared `type`\n\nis rejected with\na 400 that names it. Gemma4's native call syntax has no spelling for a\nfree-form value, and refusing is better than an unconstrained call the mapper\nmay not be able to read back. This differs from the generic JSON envelope,\nwhich can represent a free JSON value.\n\nApertus tool prompts follow\n`swiss-ai/Apertus-8B-Instruct-2509`\n\n's `chat_template.jinja`\n\nat revision\n`b946d40447b2b597999b9c86d44bee0b452c919f`\n\n: declarations are\nTypeScript under `Tool Capabilities:`\n\nin the developer turn, assistant calls\nuse `<|tools_prefix|>...<|tools_suffix|>`\n\n, and raw tool results form a bracketed\nlist inside the assistant turn that made the call. A text-plus-calls turn keeps\nthe text immediately before the call block, and an answer after the result\ncontinues that same assistant turn. The conformance gate proves these rendered\nbytes against the upstream template. No Apertus checkpoint/tokenizer was\navailable for this change, so token identity and checkpoint behavior remain\nunmeasured.\n\n`enable_thinking`\n\n, either at the top level or inside `chat_template_kwargs`\n\n,\nis the request-level form of `--think`\n\n/`--no-think`\n\n. Omitting it is not the\nsame as sending `false`\n\n: an absent field renders whatever the model family's\nown reference template renders, and that default differs per family, so\ncollapsing \"unspecified\" onto one of them would misrender the other.\n\n``` python\nimport openai\n\nclient = openai.OpenAI(\n    base_url=\"http://127.0.0.1:8080/v1\",\n    api_key=\"none\",\n)\nresponse = client.chat.completions.create(\n    model=\"runner\",\n    messages=[{\"role\": \"user\", \"content\": \"Return a status object\"}],\n    response_format={\"type\": \"json_object\"},\n)\nprint(response.choices[0].message.content)\n```\n\nConstrained buffered requests can set `choice_logprobs:true`\n\n. Decision points\nthen include legal alternatives, posterior probability over the probed legal\nset, raw logprobs, and coverage mass. `choice_logprobs_probe`\n\ndefaults to 32\nand is capped at 64; `scripts/cl-calibration.py`\n\nturns labeled records into an\nECE report.\n\nResponses requests are translated to the same prompt, sampler, and one tool\nper turn envelope as Chat Completions. Supported input includes strings and\nitem arrays, `function_call`\n\n/`function_call_output`\n\nloops, flat or nested\nfunction tools, tool choice, `text.format`\n\nfor text/JSON/schema, ordinary\nsampling controls, `max_output_tokens`\n\n, `reasoning`\n\n, and `store:false`\n\n.\n\nStreaming emits ordered typed lifecycle, text-delta, function-argument-delta,\ndone, and terminal events with monotonic `sequence_number`\n\nvalues. The\nterminal event contains usage and runner telemetry.\n\nRunner is stateless and refuses persistence or hosted-service fields rather\nthan accepting them without effect: `store:true`\n\n, `previous_response_id`\n\n,\n`background:true`\n\n, `conversation`\n\n, `truncation:\"auto\"`\n\n, `include[]`\n\n, hosted\ntools, and `parallel_tool_calls:true`\n\n.\n\nA replayed `function_call`\n\nitem must say which function it called. Runner uses\nits `name`\n\n, falls back to the sole declared function when exactly one tool is\ndeclared, and otherwise answers 400 naming `name`\n\nas the field that would fix\nit. The item is never dropped from the history: a call that silently vanished\nleft the model reading a tool result for a call it never made, with a 200 on\nthe response.\n\nA replayed `function_call`\n\nand its `function_call_output`\n\nare serialized in the\nresident model's own tool protocol - the same serializer Chat Completions uses,\nnot a generic one bolted onto this surface. A gemma-4 call comes back as\n`<|tool_call>call:NAME{...}<tool_call|>`\n\nwith gemma-4 argument formatting and\nits result as `<|tool_response>response:NAME{...}<tool_response|>`\n\n; an ornith\ncall as `<tool_call><function=NAME>…`\n\nwith the result wrapped in\n`<tool_response>`\n\n; a muse call as its `<atem:invoke>`\n\nrecipient turn with the\nresult as a named `<tool_output>`\n\n; Harmony as its `to=functions.NAME`\n\nturns.\nThe same three-turn conversation therefore renders byte-identically whether it\narrives on `/v1/chat/completions`\n\n, `/v1/responses`\n\n, or `/v1/messages`\n\n- a\ncontract pinned by goldens in `tests/test_tool_attribution.c`\n\n.\n\nMessages uses the same internal engine and constrained tool envelope. It\nsupports string or block-list system/content values, `tool_use`\n\n/`tool_result`\n\n,\nall tool-choice forms compatible with one call per turn, stop sequences,\nsampling controls, metadata, thinking-channel blocks, and Anthropic SSE event\nordering. `max_tokens`\n\nis required.\n\nRunner refuses hosted tools, MCP/container execution, image/document blocks,\nparallel tool use, `stop_sequences`\n\nsent alongside `tools`\n\n(see Chat\nCompletions above), and forced thinking on a model with no reasoning channel.\nIt implements protocol translation only; it never executes a tool.\n\nA replayed `tool_use`\n\nblock and its `tool_result`\n\nare serialized in the\nresident model's own tool protocol - the same serializer Chat Completions and\nResponses reach - so a gemma-4, ornith, or muse history is never handed the\ngeneric call syntax those models were not trained on. The result turn is named\nfrom the `tool_use`\n\nit answers (by `tool_use_id`\n\n, falling back to the sole\ndeclared tool) exactly as on the other two surfaces.\n\nA generation fault is reported as an Anthropic error object rather than a\n`Message`\n\nwith a made-up `stop_reason`\n\n- HTTP 500 `api_error`\n\nbuffered, the\ndocumented `event: error`\n\nmid-stream. See Constrained output below.\n\nClient compatibility is a dated executable observation, not something inferred\nfrom an API name. The 2026-08-03/04 sweep recorded complete local file-read\nloops for OpenCode 1.18.4, Cline CLI 3.0.46, pi 0.81.1, Continue CLI 1.5.47,\nClaude Code 2.1.220, and lean-tool-set Codex CLI 0.144.6. Aider 0.86.2 passed\ntransport/inference under `--dry-run`\n\nbut still needs a matching model edit\nprofile.\n\nCodex and other feature-rich agents can declare more than runner's 59-tool\nconstrained envelope. Disable unused app, multi-agent, and hosted-search tools\nfor a local-model session. Exact request shapes and test scope are recorded in\n[docs/agent-compatibility.md](/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/agent-compatibility.md) and\n[docs/compatibility-program.md](/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/compatibility-program.md).\n\nFor Codex CLI, configure a stateless Responses provider:\n\n```\nmodel = \"runner\"\nmodel_provider = \"runner\"\n\n[model_providers.runner]\nname = \"Xyntetik Runner\"\nbase_url = \"http://127.0.0.1:8080/v1\"\nwire_api = \"responses\"\nenv_key = \"RUNNER_API_KEY\"\nexport RUNNER_API_KEY=none\n./runner -m model.gguf --serve -c 16384\ncodex \"list the files here\"\n```\n\nCodex's system prompt and tools can consume roughly 10k input tokens before\nthe user request, so use at least a 16k context for that workflow. Runner does\nnot implement a response store; clients must send history each turn rather\nthan use `previous_response_id`\n\n.\n\nmacOS and Windows ship a menu-bar / notification-area controller. It lists\nevery runner instance live on the machine - however it was started - with the\nmodels each has loaded, and lets you stop any of them, pick a GGUF, and start\na desktop-managed server. Linux has no tray; `--tray`\n\nthere prints an honest\nerror.\n\nThe tray follows a session you sit with, and is left running afterwards so the next model can be loaded from it.\n\n| Invocation | Tray |\n|---|---|\n`runner` with no arguments at a terminal, or a double-click |\nyes |\n`runner -m model.gguf --serve` |\nyes |\n`runner -m model.gguf -i` |\nyes |\n`runner -m model.gguf -p \"...\"` |\nno |\n`--caps` , `--quantize` , `--bench-json` , `--version` |\nno |\nanything with `--no-tray` |\nno |\n| pipes, scripts, CI, Linux | no |\n\nA terminal on **either** stdin or stdout is what counts as \"a person launched\nthis\", so `runner --serve > server.log`\n\nstill raises one while CI, which\nusually has neither, does not. A one-shot `-p`\n\nrun raises nothing on purpose:\na two-second process should not leave a menu-bar icon behind it.\n\n`--tray`\n\nmeans *be* the tray rather than run a model. It is required wherever\nthere is no terminal - launchd, Task Scheduler, a service wrapper - because\nevery launch in the table above needs one. `--no-tray`\n\nopts out everywhere.\n\nOne tray runs per machine; a second exits naming the pid that owns the icon. The tray is spawned detached with its own session, so stopping a server with Ctrl-C leaves the menu bar alone, and it outlives the run that raised it.\n\nA rounded-square core with a signal motif around it. On macOS it is a template image, so it follows light and dark menu bars.\n\n| State | Glyph | Meaning |\n|---|---|---|\n| Idle | hollow core, two opposing sweeps | No runner registered. |\n| Model loaded | solid core, two opposing sweeps | A runner is up with a model resident, nothing in flight. |\n| Running | solid core, four-segment ring | Inference is in flight. |\n\nThe ring is segmented rather than closed because a menu-bar template image cannot animate: four gaps read as motion where a circle reads as a badge.\n\n`/health`\n\nalso carries what a supervisor needs to budget several runners on\none machine. `rss_bytes`\n\nis this **process's** resident set - weights, KV\ncache, activations and allocator overhead together - which is the number a\nmachine is sized against and which no per-mapping measure accounts for;\n`peak_rss_bytes`\n\nis its high-water mark. `tokens_prompt`\n\n, `tokens_generated`\n\nand `generate_seconds`\n\nare cumulative monotonic totals across every API\nsurface. `batch_steps`\n\nand `batch_sequences`\n\ncount the scheduler's microbatch\nsteps and the sequences cut into them, so `batch_sequences / batch_steps`\n\nis\nthe mean batch size over your own window; both stay `0`\n\non a server that never\nstarted continuous batching (a single slot, or swap mode). On CUDA they also\nstay `0`\n\nfor a model whose weights use a quantization the batched path has no\nbitwise-identical kernel for - a batched step must return, per sequence, the\nbits a lone step would have, so such a model decodes its sequences one at a\ntime rather than batching them into different numbers. The current CUDA\nmicrobatch loop covers gated dense transformer layers; recurrent, MoE, NoPE,\nattention-gated, and ungated xIELU models use sequential GPU forwards. Within\nthe covered family Q8_0, Q4_0, Q4_K, Q5_K, Q6_K, F32 and F16 batch; the rest do\nnot.\n\nThose are deliberately raw counters rather than a tokens-per-second field: a rate needs an averaging window, and the runner has no business choosing one for a consumer whose window differs. Difference them over your own interval. The endpoint does not count its own requests, so polling it on a timer does not show up as work.\n\n\"Loaded\" and \"running\" are told apart by `active_requests`\n\nfrom `/health`\n\n,\npolled on the same 5-second timer that refreshes the icon - so a request\nshorter than the tick can pass unseen. It is an indicator, not telemetry. When\nthe count cannot be read the icon shows \"model loaded\", because a server that\nis up but unreachable still has a model resident.\n\nConfiguration, the instance registry, autostart, uninstall, and the headless\nvalidation seams are documented in\n[docs/tray-controller.md](/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/tray-controller.md).\n\nThe macOS release is ad-hoc signed, not Apple-notarized. A browser download may\ntherefore be blocked by Gatekeeper even when its published checksum matches.\nVerify the SHA-256 checksum first, then remove the quarantine attribute from\nthe extracted binary with `xattr -d com.apple.quarantine runner`\n\n; obtaining a\nDeveloper ID and notarizing releases remains an owner action.\n\nRunner provides two sampler-level guarantees:\n\n`--json`\n\nor OpenAI`response_format.type=json_object`\n\nemits one valid JSON object.`--json-schema FILE`\n\n, OpenAI`json_schema`\n\n, Responses`text.format`\n\n, and tool parameter schemas compile to a streaming conformance validator.\n\nThe supported schema subset covers objects, arrays, strings, numbers,\nintegers, booleans, null, enums, const, type unions, numeric bounds on both\n`integer`\n\nand `number`\n\n(`minimum`\n\n/`maximum`\n\nand their exclusive forms, with a\nforced close completing the value inside the declared range), string\nlengths and supported anchored patterns, array item/count constraints,\nscalar-const `oneOf`\n\n/`anyOf`\n\n, and the tool-discriminated object union used by\nagent clients. Required properties are present, unknown properties are blocked\nfor closed objects, and tool arguments are generated against the selected\ntool's schema.\n\nObject schemas may be closed fixed-property records, unconstrained open\nobjects, or homogeneous maps: with no declared properties (or an empty\n`properties`\n\nobject), a schema-valued `additionalProperties`\n\nis enforced for\nevery arbitrary-key value. Mixed fixed properties plus open or schema-valued\nadditional properties remain unsupported and are rejected rather than\nsilently weakened.\n\nAnchored `pattern`\n\ns compile as a sequence of literal runs and repeated ASCII\nclasses (`[...]`\n\n, `\\d`\n\n, `\\w`\n\n): `^wf_[a-z0-9-]{6,}$`\n\nand `^[A-Z]{3}[0-9]{4}$`\n\nboth enforce, and a forced close mid-string completes to a string the pattern\nstill accepts. Every class before the last carries a fixed count, so which\nclass a byte belongs to follows from its offset; a variable-length class in\nthe middle is refused rather than guessed, as are `\\s`\n\n, negated classes, and\nescapes inside `[...]`\n\n.\n\nUnsupported or ambiguous constraints fail at compile/request time. In\nparticular, general overlapping `oneOf`\n\nbranches are not tracked in parallel;\nbranches must diverge at a supported discriminator. This is a subset of JSON\nSchema 2020-12, not full JSON Schema or GBNF.\n\nIf the budget ends after a document starts, runner emits the minimal legal\nsuffix and reports a length finish - on the tool-call path too: a truncated\ncall is still returned as a parseable `tool_calls`\n\nentry, but the envelope\nkeeps the truncation signal (`finish_reason: \"length\"`\n\n, Responses\n`status: \"incomplete\"`\n\nwith `max_output_tokens`\n\n, Anthropic\n`stop_reason: \"max_tokens\"`\n\n) so a caller knows the arguments are minimal\nclosures rather than the model's completed intent. If the model never starts\nthe document, runner returns empty content rather than inventing required\nvalues. Syntax and schema shape are guaranteed; semantic correctness and tool\nselection remain the model's responsibility.\n\nA client `stop`\n\nsequence is handled as a truncation the caller asked for. The\nmatched bytes are withheld from the response, as they are in unconstrained\ntext, and the constraint validator is truncated with them - re-seated on\nexactly the document the caller received - so the minimal legal suffix\ncompletes that copy rather than the longer one the model had reached. The\ndelivered document parses and conforms; `finish_reason`\n\nis `\"stop\"`\n\n(Anthropic\n`stop_reason: \"stop_sequence\"`\n\n, carrying the matched string). The suffix itself\nis never stop-matched: it is runner closing the document rather than model\ntext, and `[\"}\"]`\n\nor `[\"\\n\\n\"]`\n\nwould otherwise eat the very bytes that make it\nlegal.\n\nIf an envelope document cannot be mapped back at all, runner reports the fault\ninstead of serving the raw protocol as an answer. On the OpenAI surfaces\ncontent is empty, `finish_reason`\n\nis `\"error\"`\n\nwith\n`runner_telemetry.finish_detail: \"envelope_unmapped\"`\n\n, and Responses reports\n`status: \"incomplete\"`\n\nwith reason `envelope_unmapped`\n\n. A stream that ends\nthis way is still terminated - a terminal chunk carrying the finish reason,\nthen `data: [DONE]`\n\n, or Responses `response.incomplete`\n\n- so a client is never\nleft waiting on events that will not arrive.\n\nAnthropic Messages reports the same fault as an **error object**, not a\n`Message`\n\n. All seven of its `stop_reason`\n\nvalues describe a turn that\ncompleted, so none of them can carry a generation fault; a buffered turn\nanswers HTTP 500 with `{\"type\": \"error\", \"error\": {\"type\": \"api_error\", \"message\": ...}}`\n\n, which is the class the Anthropic SDKs retry with backoff,\nand a streamed turn - whose 200 is already sent - terminates on the protocol's\ndocumented `event: error`\n\ncarrying the same object, in place of\n`message_delta`\n\n/`message_stop`\n\n. An allocation failure during generation is\nreported the same way on both. Partial text is not returned alongside it:\nunlike a budget truncation, which is a completion and keeps its content under\n`stop_reason: \"max_tokens\"`\n\n, a fault has no `stop_reason`\n\nthat would not\nmisstate why generation stopped. `runner_telemetry.finish_detail`\n\nrides on the\nerror object so the two faults stay distinguishable.\n\nConstrained decoding pays for the sampler on every step, and several shipped\npresets - SmolLM2's, llama3's, mistral's, gpt-oss's - set `top_k = 0`\n\n, which\nmeans no truncation and a pass over the whole vocabulary. Setting `--top-k 40`\n\n(or `\"top_k\": 40`\n\nper request) measured **12–27% higher decode throughput**\nthan the same run at the preset's `top-k 0`\n\n, on an M1 with\nSmolLM2-135M-Instruct under JSON- and schema-constrained decoding.\n\nIt is an option, not a default, and it is not certified. Truncating to 40\ncandidates **changes the sampled distribution** - it is different semantics,\nnot a cheaper route to the same tokens - so it stays outside the correctness\ngates rather than becoming a preset value. Reach for it when decode throughput\nmatters more than reproducing the preset's distribution; leave it off when the\nrun is being compared against a reference. At `--temp 0`\n\nthe question does not\narise: greedy argmax bypasses `top_k`\n\n, `top_p`\n\n, `min_p`\n\nand the repeat penalty\nentirely, so the certified greedy paths are unaffected either way.\n\n`runner --caps`\n\npublishes the architecture IDs admitted by the current binary:\n\nGGUF `general.architecture` |\nNotes |\n|---|---|\n`llama` , `mistral` , `smollm` , `stablelm` |\nLlama-style dense families with family tokenizers/templates. |\n`qwen2` , `qwen3` |\nQKV-bias and per-head-QK-norm variants. |\n`qwen35` |\nDense Qwen3.5/Ornith Gated DeltaNet plus full attention; CPU and CUDA. CPU recurrent folds support speculative decode, grammar fast-forward, and exact shared-prefix restore. Any GPU-backed recurrent instance declines shared-prefix restore; a CUDA-resident recurrent layer also declines speculative decode and grammar fast-forward. |\n`qwen3moe` |\nFused and legacy split sparse-MoE layouts on CPU/CUDA; supported fused layouts on Metal. |\n`gemma3` |\nRegular and QAT layouts, sliding-window attention, sandwich norms. |\n`gemma4` |\nHeterogeneous attention, thinking channels, E-series, supported dense/MoE layouts, and the family's native tool protocol. |\n`phi3` |\nFused QKV and gate/up tensors, LongRoPE factors. |\n`gpt-oss` |\nAttention sinks, alpha-sigmoid GLU, expert biases, MXFP4 experts. Tokenizer exact (0/721 differential) and chat renders the real Harmony format (analysis channel as `reasoning_content` ) as of 2026-08-14; cross-engine greedy identity remains inside the model's own measured KV-precision sensitivity envelope rather than certified. |\n`apertus` |\nxIELU FFN; CPU and CUDA. |\n`afmoe` |\nArcee Trinity sparse MoE; CPU only. CUDA and Metal refuse it loudly as gated attention plus sparse MoE, rather than misreporting a quantization problem. |\n`muse-glimmer` |\nMeta Muse Glimmer 30B, text path: gated attention, QK and sandwich norms, SWA with NoPE globals, softcapped logits. CPU, CUDA and Metal. Certified; evidence in `docs/muse-glimmer-cert-2026-08-11.md` and `docs/muse-atem-cert-2026-08-11.md` . No vision encoder. Native atem definitions/results, recipient-constrained generation, truncation recovery, multi-call mapping, and buffered/SSE parsing are implemented and selected automatically for tool requests. |\n`granite` |\nIBM Granite dense (3.x/4.1): the four muP scalars (embedding, fixed attention, residual, divided logit). CPU, CUDA and Metal. Certified; evidence in `docs/granite-cert-2026-08-11.md` . granitemoe is a separate arch id and not admitted; granitehybrid is admitted separately, below. |\n`granitehybrid` |\nGranite-4 h-series: a Mamba-2 selective-SSD recurrence (causal conv1d + the input-dependent state-space scan, with the gated RMS norm) interleaved with GQA attention, the layer type read per-layer from the `attention.head_count_kv` array (0 ⇒ recurrent); the attention layers are NoPE (`rope.scaling.finetuned=false` ); the four granite muP scalars. Both published FFN layouts are supported: dense h-micro has a gated MLP on every layer and runs on CPU and CUDA; sparse h-small has a routed MoE FFN plus an always-on shared expert and currently runs on CPU because those two branches have no device path. The dense h-micro CUDA path is CPU-token-identical over 600/600 greedy tokens with per-run mean |\n`nemotron_h` |\nNVIDIA Nemotron-H (Nemotron-Nano-9B-v2): a Mamba-2 selective-SSD recurrence interleaved with GQA attention and dense MLP blocks, where each block is EXACTLY ONE of three kinds (SSM | attention | MLP), typed per-layer off `attention.head_count_kv` (0) and `feed_forward_length` (0). NON-MoE and no muP scalars - unlike granitehybrid; the MLP is a gate-less squared-ReLU FFN (`down(relu(up(x))^2)` ), attention is NoPE (`rope.scaling.finetuned=false` ), and the SSM uses a GROUPED scan (`ssm.group_count=8` ): B/C are shared across groups of heads and broadcast (group g covers heads [g·H/G, (g+1)·H/G)) - the same grouped scan `nemotron_h_moe` (Nemotron-3.5 Lightning) also uses - here first proven WITHOUT MoE, and admitted WITH MoE in the row below. CPU and CUDA: the Mamba-2 SSD scan, causal conv1d, gated RMS norm and squared-ReLU FFN all have device kernels, and full 56-layer offload is greedy byte-identical to the CPU path on the real Nano-9B-v2 Q8_0 (3 prompts x 48 decode steps plus an 88-token multi-tile prefill; re-verified independently post-merge at 32 tokens). Device prefill currently runs the per-token loop (correct, unoptimized); no Metal SSM path. Verified against llama.cpp b10353 on the real Nemotron-Nano-9B-v2 at Q8_0 (same GGUF both engines, CPU): 5/6 greedy completions byte-identical (including both 256-token generations); the sole miss is a single-token near-tie where both engines share the same top-3 candidates and llama.cpp's own top-1/top-2 gap is ~0.075 nats (an FP-summation-order coin-flip), i.e. the quantisation noise floor, not wrong math. Chunked-scan prefill (the grouped scan tiled into chunks, parallel across heads within a chunk, SSD state + conv ring carried across chunk boundaries), bit-identical to the serial per-token sweep and pinned chunked == serial in `make test` ; the recurrent-state cache seam is wired (fold snapshotted/restored on rewind, and stored beside the KV so an exact CPU prompt-prefix hit restores it in a memcpy). CPU speculative decode and grammar fast-forward use the per-round fold checkpoint; full GPU offload and partial splits with a CUDA-resident recurrent layer decline them. |\n`nemotron_h_moe` |\nNVIDIA Nemotron-3.5-Lightning-30B-A3B: `nemotron_h` with the dense squared-ReLU MLP replaced by a gate-less squared-ReLU MoE (128 experts / 6 used, no gate branch) plus an always-on gate-less shared expert; the router reuses the general softmax/group/scale/norm path. Same three-way block typing, grouped scan (`n_group=8` ), and NoPE attention as `nemotron_h` . Runs on CPU: the SSM scan has a device kernel, but this family's router (weight-normed, scaled) and gate-less shared expert have no device path, so the backend falls back to CPU there. Greedy vs llama.cpp `ea12b27` on the real Lightning-30B Q4_0 (CPU, 8 tok × 5 prompts): 4/5 byte-identical, the one divergence a near-tie on an open-ended counting continuation (noise floor, not wrong math - the coherent `Paris. … Berlin.` completion matches exactly). Evidence: `docs/compat-reports/ssm-greedy-reference-2026-08-20/` . |\n\nAdmission remains layout-specific: an unsupported split expert layout, a\nnon-SiLU MoE outside gemma-4's dual-branch form, or an architecture-specific\ntensor arrangement is refused even when the architecture ID is listed. The\nalways-on shared expert (Qwen2-MoE/DeepSeek form: a dense FFN over the same\nnormed input, summed with the routed output, optionally gated) is **supported**\nand is what afmoe uses; its width and tensors are shape-checked at load, and\n`expert_shared_count`\n\nset without the tensors present is an error rather than a\nsilently dropped branch.\n\n| Area | Current support |\n|---|---|\n| File format | GGUF v2/v3, mmap/file-mapped host weights, including standard local multi-part sets. |\n| Tokenizers | SPM and byte-level BPE with llama, qwen2/qwen35, smollm, afmoe, tekken, llama4/gpt-4o, Gemma, and GPT-2-family pre-tokenization rules. |\n| Quantizations | `--caps` lists the admitted tensor formats: the k-quant and legacy families plus MXFP4 and the codebook i-quants (IQ1_S/M, IQ2_XXS/XS/S, IQ3_XXS/S, IQ4_NL/XS). The IQ1, IQ2 and IQ3 families are CPU-only with NEON/AVX2 dequant kernels; CUDA and Metal refuse them loudly, naming the exact tensor and type that caused the CPU fallback. |\n| Transformer | RMSNorm, adjacent-pair and NeoX RoPE, grouped-query attention, SwiGLU/GELU/xIELU family paths, tied embeddings, dense and selected sparse MoE. |\n| Sampling | Greedy, temperature, top-k, top-p, min-p, repeat penalty, stop strings, JSON/schema constraints, speculative decoding. |\n| Context | Batched prefill, f16/q8 KV, linear/YaRN/llama-3 scaling, automatic extension. |\n| Serving | Chat Completions, Responses, legacy completions, embeddings, Anthropic Messages, SSE, parallel slots, model swap, prefix reuse. |\n| Desktop | macOS menu bar and Windows notification-area controller. |\n\nNot implemented: Vulkan; TLS/auth; remote bind; remote/streamed GGUF parts; the\n`qwen2moe`\n\n/`deepseek2`\n\n/`kimi`\n\narchitecture IDs (their shared-expert *layout* is\nimplemented, as above - the architectures are not admitted) or MLA attention;\nMamba/Jamba; MTP/NextN draft-head consumption (those tensors load and are\nskipped, so dense decoding is unchanged); full GBNF; image/document inputs;\nhosted tools; response persistence; or parallel tool calls on the Responses and\nMessages surfaces (Chat Completions supports it, buffered and streaming).\n\nThe machine-readable manifest is\n[ tests/compatibility/models.json](/Joakimpalm-Zen/xyntetik-runner/blob/main/tests/compatibility/models.json). It pins\nfiles by SHA-256 and declares checks independently:\n\n| Check | Meaning |\n|---|---|\n`load` |\nThe pinned file hashes and loads. |\n`tokenizer` |\nThe committed 721-string corpus is compared with the model's Hugging Face tokenizer; revision-bound ID captures make declared rows replayable offline. |\n`greedy_reference` |\nGreedy tokens are compared with a pinned llama.cpp revision. |\n`cpu_cuda` |\nCPU and CUDA scalar-path greedy output are compared. |\n`chat` |\nA real Chat Completions request answers through the model template. |\n`tool` |\nA function call round-trips as schema-conformant tool output. |\n`long_context` |\nA needle is retrieved from an extended context. |\n\nBeing present in the manifest does not mean every check passed. Read each entry's declared checks and notes. Current high-signal caveats include:\n\nEvery release ships a schema-versioned report under `docs/compat-reports/`\n\n,\nand `scripts/check-release.py`\n\nenforces it: a tag whose version has no\n`<version>-<date>.json`\n\nin that directory fails the release check rather\nthan shipping with an unpublished ledger. Generate it against the pinned\nfiles available on the release box with:\n\n```\npython3 scripts/compat_matrix.py --models-root /path/to/models \\\n  --runner ./runner --reference /path/to/llama-server \\\n  --verify-files --execute-checks --out docs/compat-reports/<release>-<date>.json\n```\n\nThe executable classes are SHA/load, tokenizer differential when the manifest\ndeclares a reference and the corpus exists, and greedy reference when both\nbinaries and the pinned model are present. Every other declared check, and any\ncheck missing a prerequisite, is retained as `not_executed`\n\nwith a machine-\nreadable reason; absence from a run is never presented as a pass.\n\nTokenizer rows with `tokenizer_reference_ids`\n\nuse the committed capture rather\nthan the network. Each capture names the Hugging Face repository and immutable\nrevision, binds itself to the corpus SHA-256, and contains only token IDs - never\ncredentials or model weights. `scripts/difftok.py --ref-ids CAPTURE`\n\nis the\nstandalone replay path; `--capture CAPTURE --ref-revision COMMIT`\n\ncreates one\nduring an authenticated evidence run.\n\n- Qwen3-4B's 2026-08-03 scalar CPU/CUDA recheck passed only 4 of 5 prompts;\n**re-measured 2026-08-20 with the current gate: 9/9 prompts byte-exact at 128 tokens, zero near-ties**(the intervening router-bias, tensor-core-identity and canonical-quantizer fixes resolved it). Per-row evidence:`docs/compat-reports/cpu-cuda-128/qwen3-4b-q4km-2026-08-20.json`\n\n. - Canonical gpt-oss-20b passed an earlier 5-of-5, 16-token partial-offload test\non an RTX 3070, but failed CPU/CUDA identity and chat/tokenizer gates on the\nlater Blackwell full-offload matrix. Hardware and test-contract scope matter.\nIts tokenizer differential is exact (0/721) and its\n**chat** gate now passes too: gpt-oss renders through a real Harmony template as of 2026-08-14 (`<|channel|>`\n\n-structured turns,`<|return|>`\n\nas the stop), where it previously fell through to llama2's`[INST]`\n\nmarkup and ran away. The analysis channel is suppressed from`content`\n\nand surfaced as`reasoning_content`\n\n;`enable_thinking: false`\n\nskips it. Harmony tool calling uses the model's native commentary/recipient protocol and strict declared- argument constraints. Measured transcripts:[docs/gpt-oss-harmony-2026-08-14.md](/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/gpt-oss-harmony-2026-08-14.md). The CPU/CUDA identity row was**re-measured on 2026-08-19** now that the file is on the Blackwell box, after the 2026-08-18 router-bias fix ([docs/cuda-gptoss-router-bias-2026-08-18.md](/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/cuda-gptoss-router-bias-2026-08-18.md)).`gpt-oss-20b-MXFP4`\n\nat**full offload**- the deployment configuration - passes`test-gpu-identity`\n\nat 0.000732 of logit range against the 2e-3 bound. The bound is exceeded only under*partial*offload (0.00356 at 1 GPU layer), where a single device layer's reduction-order rounding is amplified through the remaining CPU layers: the divergence is non-monotonic in GPU-layer count, a shape a systematic wrong op cannot produce. The mechanism is discrete top-4-of-32 expert-routing chaos (two experts tied to four decimals reorder under a sub-ULP perturbation), and the model already disagrees with itself on 3 of 16 prompts under a CPU-only KV-precision change. No CUDA correctness defect remains; gpt-oss is gated at its measured sensitivity floor, not at dense-model logit identity.`gpt-oss-120b-MXFP4`\n\n(63 GB, unable to fully offload on a 24 GB MIG) reproduces 0.00245 at 4 GPU layers - the same amplification effect at greater depth. Full bisection:[docs/cuda-gptoss-divergence-2026-08-19.md](/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/cuda-gptoss-divergence-2026-08-19.md). - Gemma-4-26B-A4B QAT's old 16-token CPU/CUDA result is not a substitute for the manifest's pending 128-token re-verification.\n- Numerically sensitive models may use a measured self-sensitivity floor instead of claiming cross-engine token identity.\n\nThe full 2026-08-05 pass/fail/refusal matrix, including failed derivatives, is\nin [docs/cert-matrix-status.md](/Joakimpalm-Zen/xyntetik-runner/blob/main/docs/cert-matrix-status.md). Architecture and\nmodel-family additions must update the manifest and executable gates, not only\nthis README.\n\n```\nsrc/gguf.c            GGUF metadata and tensor-table parser\nsrc/tokenizer.c       SPM/BPE tokenization and family pre-tokenizers\nsrc/quants.c          scalar, AVX2/FMA, and NEON quantized dot kernels\nsrc/model.c           tensor admission, weight wiring, and forward pass\nsrc/sample.c          sampling filters and token selection\nsrc/jsonmode.c        incremental JSON-prefix validation\nsrc/schema.c          JSON-Schema compiler and streaming validator\nsrc/template.c        chat templates, thinking channels, and tool syntax\nsrc/engine.c          prompt feeding, prefix cache, constrained sampling, speculative decode\nsrc/quantize.c        requantization and stacked-MoE expert pruning\nsrc/scheduler.c       persistent worker scheduling\nsrc/cuda.c            CUDA driver backend; kernels.cu becomes embedded PTX\nsrc/metal.m           Metal backend; kernels.metal is embedded at build time\nsrc/server.c          loopback HTTP server, slots, routing, and lifecycle\nsrc/completion.c      shared completion request/response path\nsrc/api_responses.c   OpenAI Responses translation\nsrc/api_anthropic.c   Anthropic Messages translation\nsrc/registry.c        model swap and unload lifecycle\nsrc/vramreg.c         cross-process VRAM ownership and bounded waiting\nsrc/tray*.c           macOS/Windows desktop controller\nsrc/compat.c          platform process, memory, mmap, clock, and socket helpers\nsrc/main.c            CLI parsing, utility modes, and --caps\npython/               supported Python client and subprocess integration\n```\n\nHost weights remain quantized in the mapped GGUF and are dequantized while\ncomputing. CPU memory is approximately mapped weights plus KV and scratch.\nCUDA copies selected weights and compute/KV buffers to VRAM; Metal wraps mapped\nweights in unified memory. The load log and `--caps`\n\nare the sizing sources for\nan exact model/machine combination.\n\nXyntetik Runner is developed independently. If it is useful to you and you\nwant to help fund the hardware and measurement time behind it, you can\n[support the project here](https://buy.stripe.com/9B69AUddpdx9auHgP27N600).", "url": "https://wpnews.pro/news/xyntetik-runner-a-gguf-runtime-in-plain-c-that-proves-what-it-runs", "canonical_source": "https://github.com/Joakimpalm-Zen/xyntetik-runner", "published_at": "2026-08-25 17:46:08+00:00", "updated_at": "2026-08-25 18:16:01.797465+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "developer-tools", "ai-infrastructure"], "entities": ["Xyntetik Runner", "Joakimpalm-Zen", "IBM Granite", "llama.cpp", "Apache 2.0"], "alternates": {"html": "https://wpnews.pro/news/xyntetik-runner-a-gguf-runtime-in-plain-c-that-proves-what-it-runs", "markdown": "https://wpnews.pro/news/xyntetik-runner-a-gguf-runtime-in-plain-c-that-proves-what-it-runs.md", "text": "https://wpnews.pro/news/xyntetik-runner-a-gguf-runtime-in-plain-c-that-proves-what-it-runs.txt", "jsonld": "https://wpnews.pro/news/xyntetik-runner-a-gguf-runtime-in-plain-c-that-proves-what-it-runs.jsonld"}}