Crystal ML Library: Autograd, Tensors, Neural Networks, Optimizers

Cogni-ML, a Crystal machine learning library with native Apple Silicon GPU acceleration, has been released. It provides tensors, autograd, neural network layers, optimizers, and native Metal inference for GGUF models including Qwen 3.5 and nomic-embed-text-v2-moe. The library targets production-oriented AI workloads on Apple Silicon with quantized kernel support and speculative decoding.

Crystal machine learning library with native Apple Silicon GPU acceleration. Cogni-ML is currently two things: - A general Crystal ML toolkit: tensors, autograd, NN layers, optimizers, GGUF readers, and llama.cpp bindings. - A native Metal inference lab for GGUF models, with production-oriented work on nomic-embed-text-v2-moe embeddings and Qwen 3.5 text generation. - Native Metal embedding pipeline for nomic-embed-text-v2-moe . - Native Qwen 3.5 9B GGUF inference path for Apple Silicon Metal. - Q4 K/Q5 K/Q6 K/Q8 0 quantized matmul kernels. - Chunked Qwen 3.5 prefill, decode wave scheduling, prompt-state cache restore, and exact speculative decode harnesses. - ComputeGraph wave scheduling with offset-aware barrier optimization. - Crystal autograd engine, NN layers, and Adam/AdamW optimizers. - llama.cpp FFI bindings for general GGUF model access. src/ml/ core/ Tensor, Shape, MetalBuffer autograd/ Variable, GradFn backward pass nn/ Linear, LayerNorm, MultiHeadAttention, ViT optim/ Adam/AdamW llm/ llama.cpp FFI bindings gguf/ GGUF reader, tokenizer, dequantization, Qwen35, NomicBertMoE metal/ Device, ComputeEncoder, ComputeGraph, GraphEncoder The native Qwen path targets Qwen3.5-9B-Q4 K M.gguf on Apple Silicon. The code supports: - Qwen 3.5 GGUF metadata and tokenizer loading. - Q4 K, Q5 K, Q6 K, and Q8 0 quantized projections. - Full-attention layers with GQA, partial RoPE, KV cache writes, and fused output projection. - DeltaNet/recurrent layers with GPU-resident recurrent state and chunked prefill scan. - Chunked prefill with final-token top1 shortcut. - Decode wave scheduling to reduce command-buffer boundaries. - Native Qwen BPE tokenizer with an external llama-tokenize fallback for A/B. - Exact prompt-state save/restore, tokenized-prompt reuse, and longest-prefix prompt cache. - Exact speculative decode harnesses: - neural draft with Qwen 3.5 0.8B Q8 0, - n-gram/cache draft for repeated/generated-template text, - target-verifier chunks with row-batched top1 for larger accepted chunks. The 9B Q4 K M path is the primary verified target. Qwen 3.6 27B is a scale-up target, but it should be treated as experimental until local correctness and performance runs are completed. The developer CLIs default to local LM Studio / llama.cpp-style paths: ~/.cache/lm-studio/models/lmstudio-community/Qwen3.5-9B-GGUF/Qwen3.5-9B-Q4 K M.gguf ~/.cache/lm-studio/models/lmstudio-community/Qwen3.5-0.8B-GGUF/Qwen3.5-0.8B-Q8 0.gguf ~/SrcArchives/AI/llama.cpp/build/bin/llama-tokenize ~/SrcArchives/AI/llama.cpp/build/bin/llama-bench Most benchmark/probe CLIs also accept --model , --target , --draft , --tokenizer-bin , or environment overrides. bin/qwen35 generate.cr is intentionally a small demo and currently uses its constants at the top of the file. Build the CPU-only GGUF/Qwen metadata smoke on Linux, CUDA hosts, or any environment where Metal is unavailable: crystal build -Dcpu only bin/qwen35 gguf info.cr -o build/qwen35 gguf info ./build/qwen35 gguf info --model /path/to/Qwen3.5-9B-Q4 K M.gguf ./build/qwen35 gguf info --model /path/to/Qwen3.5-0.8B-Q8 0.gguf --load-weights This entrypoint intentionally does not run inference. It verifies GGUF parsing, Qwen 3.5/3.6 hparams, tensor inventory, and the structured Qwen35Weights loader without pulling the Metal bridge into a Linux build. Build the minimal Crystal CUDA Driver API smoke on NVIDIA/Linux hosts: crystal build bin/cuda driver smoke.cr -o build/cuda driver smoke ./build/cuda driver smoke 4096 The CUDA smoke is a backend boundary probe only: it links libcuda , loads embedded PTX, launches a vector-add kernel, and checks the result. CUDA probe code uses src/ml/cuda/driver.cr for reusable CUDA context, module/function, launch, copy, synchronize, and device-buffer ownership. This is intentionally small: it owns the raw CUDA Driver API lifecycle and calls, while higher-level layer execution is still probe-local until the CUDA backend split is promoted. It also provides ML::CUDA::ResidentSequenceRunner , a thin lifecycle facade for resident sequence probes with explicit upload weights , reset sequence , run sequence , and read outputs phases. src/ml/cuda/qwen recurrent layer runner.cr is the first Qwen-specific runner extraction: it owns one recurrent layer's CUDA modules, device buffers, kernel parameters, weight upload, sequence reset, token launch graph, and output readback. QwenRecurrentLayerRunner::Weights.load owns GGUF tensor lookup, tensor-shape/type validation, and raw weight reads for the runner, including recurrent-layer ffn down tensors stored as either Q4 K or Q6 K. CPU-reference comparison intentionally remains in the probe. Build the first quantized CUDA correctness probe on NVIDIA/Linux hosts: crystal build -Dcpu only bin/cuda q8 gemv probe.cr -o build/cuda q8 gemv probe ./build/cuda q8 gemv probe \ --model /path/to/Qwen3.5-0.8B-Q8 0.gguf \ --tensor blk.0.ffn up.weight \ --kernel warp4 \ --reps 100 \ --warmup 10 cuda q8 gemv probe loads a real GGUF Q8 0 tensor, launches a Crystal-driven CUDA Driver API GEMV kernel over the raw GGUF block layout, and compares against the existing CPU QuantMatmul reference. --kernel scalar keeps the first one-thread-per-output-row correctness kernel; the default --kernel warp4 maps four output rows to four warps per thread block and is the current faster probe shape. This is still a standalone backend-boundary probe, not an optimized Qwen CUDA inference path yet. The current full qwen35 generate CLI remains Metal-first. Build the first Q4 K CUDA correctness probe for Qwen 9B/27B-style target tensors: crystal build -Dcpu only bin/cuda q4k gemv probe.cr -o build/cuda q4k gemv probe ./build/cuda q4k gemv probe \ --model /path/to/Qwen3.5-9B-Q4 K M.gguf \ --tensor blk.0.attn gate.weight \ --kernel warp4 \ --reps 20 \ --warmup 3 cuda q4k gemv probe uses the raw GGUF Q4 K block layout d , dmin , 12-byte packed scales/mins, 128-byte packed nibbles and checks the CUDA output against the CPU QuantMatmul Q4 K reference. --kernel scalar keeps the first correctness kernel; the default --kernel warp4 maps four output rows to four warps per block and is the current faster probe shape. Build the Q6 K CUDA correctness/speed probe for Q4 K M tensors that remain in Q6 K: crystal build -Dcpu only bin/cuda q6k gemv probe.cr -o build/cuda q6k gemv probe ./build/cuda q6k gemv probe \ --model /path/to/Qwen3.5-9B-Q4 K M.gguf \ --tensor blk.0.ffn down.weight \ --kernel warp4 \ --reps 10 \ --warmup 2 cuda q6k gemv probe covers the GGUF Q6 K block layout ql , qh , signed scales, d used by output/value/down projections in mixed-quant target models. Like the Q4 K/Q8 0 probes, it is a standalone backend primitive check; full CUDA Qwen execution is still a separate backend split. Build the first GPU-resident FFN sequence probe: crystal build -Dcpu only bin/cuda ffn sequence probe.cr -o build/cuda ffn sequence probe ./build/cuda ffn sequence probe \ --model /path/to/Qwen3.5-9B-Q4 K M.gguf \ --layer 0 \ --reps 10 \ --warmup 2 cuda ffn sequence probe composes the checked CUDA primitives as Q4 K ffn gate + Q4 K ffn up - SwiGLU - Q6 K ffn down while keeping the input, intermediate activations, and output projection input GPU-resident. Only the final hidden vector is copied back for comparison against the CPU QuantMatmul FFN reference. Build the full-attention input projection bundle probe: crystal build -Dcpu only bin/cuda attn projection probe.cr -o build/cuda attn projection probe ./build/cuda attn projection probe \ --model /path/to/Qwen3.5-9B-Q4 K M.gguf \ --layer 3 \ --reps 10 \ --warmup 2 cuda attn projection probe runs Q4 K attn q + Q4 K attn k + Q6 K attn v from one GPU-resident hidden vector and copies Q/K/V back only after all projections complete. It targets full-attention layers such as blk.3 in Qwen3.5 9B. The probe now routes through ML::CUDA::QwenFullAttnProjectionRunner , supports --tokens N , and keeps Q/K/V outputs GPU-resident until the final correctness readback. It is the reusable input-projection boundary for future full-attention/KV CUDA work, not a complete full-attention layer runner yet. Build the full-attention Q/K normalization + RoPE + KV-cache boundary probe: crystal build -Dcpu only bin/cuda full attn kv probe.cr -o build/cuda full attn kv probe ./build/cuda full attn kv probe \ --model /path/to/Qwen3.5-9B-Q4 K M.gguf \ --layer 3 \ --tokens 4 \ --start-pos 2 \ --max-seq 12 cuda full attn kv probe now routes through ML::CUDA::QwenFullAttnLayerRunner , a residual-hidden-to-final-hidden wrapper around the projection runner and ML::CUDA::QwenFullAttnKVRunner . The projection runner can apply the initial attn norm on CUDA from residual hidden states before Q/K/V projection; Q is split into normalized/RoPE'd Q and gate, K is RMSNormed/RoPE'd, K/V rows are appended to a CUDA-resident cache at start pos , a correctness-first serial CUDA kernel computes GQA scores, softmax, value reduction, and Q-gate multiplication, the resident gated attention output is projected through attn output.weight , and the layer tail runs residual add, post-attention RMSNorm, FFN gate/up/SwiGLU/down, and final residual. It checks Q, gate, K, gated attention output, projected attention output, final hidden, K-cache, and V-cache against the CPU Qwen reference. This is now a clean one-layer semantics probe with device input/output hooks for mixed-stack composition, but it is not yet an end-to-end Linux decode path: full/recurrent stack scheduling, logits/top1, tokenizer/sampling, restored nonzero prefix KV, and a faster attention kernel remain separate gates. Build the Q5 K CUDA recurrent-QKV probe: crystal build -Dcpu only bin/cuda q5k gemv probe.cr -o build/cuda q5k gemv probe ./build/cuda q5k gemv probe \ --model /path/to/Qwen3.5-9B-Q4 K M.gguf \ --tensor blk.0.attn qkv.weight \ --reps 10 \ --warmup 2 cuda q5k gemv probe covers the GGUF Q5 K block layout used by recurrent-layer combined attn qkv.weight tensors in the current Qwen3.5 9B Q4 K M file. Build the recurrent-layer projection bundle probe: crystal build -Dcpu only bin/cuda recurrent projection probe.cr -o build/cuda recurrent projection probe ./build/cuda recurrent projection probe \ --model /path/to/Qwen3.5-9B-Q4 K M.gguf \ --layer 0 \ --reps 10 \ --warmup 2 cuda recurrent projection probe runs Q5 K attn qkv + Q4 K attn gate + Q4 K ssm alpha + Q4 K ssm beta from one GPU-resident hidden vector and copies the four outputs back only after all kernels complete. It is the first CUDA recurrent projection-bundle proof; DeltaNet recurrence, convolution, state updates, and ssm out remain separate work. Build the synthetic DeltaNet output slice probe: crystal build -Dcpu only bin/cuda deltanet output probe.cr -o build/cuda deltanet output probe ./build/cuda deltanet output probe \ --model /path/to/Qwen3.5-9B-Q4 K M.gguf \ --layer 0 \ --reps 10 \ --warmup 2 cuda deltanet output probe runs a synthetic CUDA DeltaNet state update, applies post RMSNorm/SiLU gating on GPU, and feeds the result directly into the real Q4 K ssm out.weight projection. It is a stateful boundary probe, not a full recurrent layer: recurrent conv prep, alpha/beta transforms, residuals, and FFN remain separate work. Build the recurrent prep/output slice probe: crystal build -Dcpu only bin/cuda recurrent prep output probe.cr -o build/cuda recurrent prep output probe ./build/cuda recurrent prep output probe \ --model /path/to/Qwen3.5-9B-Q4 K M.gguf \ --layer 0 \ --tokens 4 \ --reps 10 \ --warmup 2 cuda recurrent prep output probe now composes one full recurrent layer token slice: input RMSNorm, real recurrent projection bundle attn qkv , attn gate , ssm alpha , ssm beta , recurrent conv prep, alpha/beta transforms, DeltaNet, post RMSNorm/SiLU, Q4 K ssm out , residual add, post-attention RMSNorm, Q4 K FFN gate/up, SwiGLU, Q6 K FFN down, and final residual. --tokens N runs a GPU-resident sequence through persistent conv/SSM state and compares all token outputs plus final recurrent states against the CPU reference. The probe now separates one-time weight upload from per-sequence input/state reset and prints weight upload ms ; timed cuda ms per token excludes the persistent weight upload. GGUF recurrent-layer tensor loading is routed through QwenRecurrentLayerRunner::Weights.load , so the probe no longer manually passes every raw tensor into the runner constructor. It is still a standalone one-layer probe, not an end-to-end Linux decoder. Build the recurrent multi-layer stack scaffold: crystal build -Dcpu only bin/cuda recurrent stack probe.cr -o build/cuda recurrent stack probe ./build/cuda recurrent stack probe \ --model /path/to/Qwen3.5-9B-Q4 K M.gguf \ --layers 0,2,4 \ --tokens 2 cuda recurrent stack probe chains multiple QwenRecurrentLayerRunner instances and compares the final hidden sequence plus each layer's recurrent conv/SSM state against the CPU reference. The default path hands each recurrent layer's CUDA output buffer directly to the next layer's CUDA input; --host-handoff keeps the older host-copy route as a debug oracle. This is still a recurrent-only scaffold, not an end-to-end Linux decoder. Build the mixed recurrent/full-attention CUDA stack probe: crystal build -Dcpu only bin/cuda mixed stack probe.cr -o build/cuda mixed stack probe ./build/cuda mixed stack probe \ --model /path/to/Qwen3.5-9B-Q4 K M.gguf \ --layers 0,1,2,3,4 \ --tokens 2 \ --start-pos 2 \ --max-seq 12 cuda mixed stack probe composes QwenRecurrentLayerRunner and QwenFullAttnLayerRunner in model layer order with device-resident hidden handoff across recurrent/full-attention boundaries, then runs QwenOutputHeadRunner for output RMSNorm, quantized lm-head projection, and resident top1. The layer/head loop is now owned by ML::CUDA::QwenMixedStackRunner , which is the first model-slice decode-state object for CUDA. By default the probe copies back only the CUDA top1 id/value plus hidden/state debug outputs; pass --read-logits to also copy full logits for attribution, and --profile-phases to insert per-layer/head synchronizations and print attribution lines. It compares the final hidden sequence, top1, recurrent conv/SSM states, and full-attention KV cache rows against the CPU reference. This is the first mixed-stack CUDA correctness scaffold through resident top1; it still stops before tokenizer/sampling, repeated full-model decode ownership, and an optimized topK/sampling kernel. The current resident top1 is a simple two-phase partial-scan/reduce kernel: correct, but not yet promoted as a speed-optimized head. Experimental CUDA exact cache-replay fast lane: crystal build --release --no-debug -Dcpu only \ bin/cuda mixed stack probe.cr \ -o build/cuda mixed stack probe ./build/cuda mixed stack probe \ --model /path/to/Qwen3.5-9B-Q4 K M.gguf \ --all-layers \ --max-seq 256 \ --greedy-loop-tokens 64 \ --greedy-loop-probe-chunk-gamma 16 \ --greedy-loop-probe-chunk-active-verify \ --greedy-loop-probe-ngram \ --greedy-loop-probe-ngram-source-history "$SOURCE TOKEN IDS" \ --greedy-loop-probe-ngram-replay-start 1 \ --greedy-loop-probe-ngram-cursor-only \ --greedy-loop-probe-ngram-trusted-source \ --greedy-loop-probe-ngram-schedule 64 \ --skip-debug-readback This path is exact verification of a validated source/cache cursor, not a general-purpose sampler. It first checks that the live prefix matches the source history at the replay cursor. If the prefix gate fails, the active verifier path is disabled before any proposals are trusted and the probe falls back to plain greedy target decode. If the gate passes, proposal chunks are verified through the same resident CUDA stack and rejected chunks restore the exact target state. Use smaller schedules such as 4,4,8,16 for weaker proposal sources where early reject economics matter; use bulk schedules such as 64 only for artifact-level trusted replay where the source history was produced by the same model/cache contract and every proposed token is still target-verified before commit. Trusted artifact restore lower-bound: ./build/cuda mixed stack probe \ --model /path/to/Qwen3.5-9B-Q4 K M.gguf \ --all-layers \ --max-seq 256 \ --known-replay-history "$SOURCE TOKEN IDS" \ --known-replay-start 1 \ --known-replay-tokens 64 \ --known-replay-trusted-artifact-restore \ --skip-debug-readback For a conservative host-backed artifact simulation, add: --known-replay-trusted-artifact-host-restore To snapshot only KV rows up to the restored cursor instead of the full max seq cache, use: --known-replay-trusted-artifact-live-kv To include diagnostic artifact file IO and hash timing, add: --known-replay-trusted-artifact-io-probe \ --known-replay-trusted-artifact-io-path /path/to/artifact.bin This is a different contract from verified replay. It simulates a cache artifact that already contains the exact token span and the post-span decode state for the same model/config/source hash. The timed region restores that state and emits cached tokens; it does not recompute the verifier body. Use it only for session/cache artifacts whose model hash, tokenizer hash, prefix hash, token span, and state artifact hash have been validated. The default trusted-artifact probe snapshots/restores state inside device memory and is a pure lower bound. The host-backed variant copies the complete decode state to host memory, poisons the runner, then times host-to-device restore. On the RTX 5060 Ti Qwen3.5-9B snapshot with max seq=128 , that state is 61,079,552 bytes; restore costs 12.259ms for a 64-token artifact and 12.331ms for a non-zero start9/24-token artifact. This is still not full production IO: durable lookup, disk reads, hashing, and live-length KV trimming remain outside the measured region. The live-KV variant preserves exact output while reducing only the full-attention cache portion. On the same host/model, start1/gen64 stores 56,885,248 bytes and restores in 11.358ms ; start9/gen24 stores 54,788,096 bytes and restores in 10.984ms . The modest delta is a useful finding: recurrent DeltaNet state, not KV capacity, dominates this short-context artifact. On the persistent reefy storage path, the live-KV start1/gen64 artifact breaks down into 52,690,944 recurrent-state bytes and 4,194,304 KV bytes. The diagnostic file path measured write 17.695ms , read 42.046ms , SHA-256 verification 59.556ms , and H2D restore 13.003ms . For start9/gen24, the artifact is 52,690,944 recurrent bytes plus 2,097,152 KV bytes; write/read/ hash/restore measured 18.249/42.994/53.814/12.825ms . The SHA-256 timing uses the host sha256sum / shasum tool to avoid adding OpenSSL linkage to the CUDA probe, so treat it as a product-boundary diagnostic rather than a kernel metric. An experimental contiguous-read reconstruction reduced start1/gen64 read time from 42.046ms to 36.310ms while preserving 64/64 exact output, but start9/ gen24 remained noisy 43.618ms read . This points to some avoidable allocation/ fragmentation cost, but not enough to change the main conclusion: cache artifacts need async prefetch or a resident artifact service to stay off the decode critical path. A first scalar CPU recurrent-state block-INT8 codec diagnostic compresses the 52,690,944 recurrent bytes to 13.18-14.00MB 25.0-26.6% depending on block size. On start1/gen64, block sizes 64/256/1024/4096 measured relative RMSE 0.008798/0.012300/0.015476/0.019087 ; encode/decode stayed around 457-469ms / 121-125ms . A follow-up restore gate now decodes the block-INT8 recurrent buffers back into a host snapshot, restores that state, and runs one continuation token against an exact uncompressed restored state. On the RTX 5060 Ti host, block sizes 64/256/1024/4096 preserved the same next top1 on start9/gen24, and block 256/4096 also preserved the same next top1 on a later start33/gen8 cursor. The multi-step source-aligned gate is stronger: block4096 preserved 16/16 continuation top1 ids on start9/gen24, and block256/block4096 preserved 8/8 continuation top1 ids on start33/gen8. The free-run greedy gate now starts both exact-restored and decoded-INT8-restored states from the same token and feeds back each path's own generated top1; block256 preserved 8/8 free-run ids on start33/gen8, and block4096 preserved 16/16 free-run ids on start9/gen24. A wider six-cursor free-run sweep changed the conclusion: block256 preserved 16/16 parity on all six tested cursors, and block1024 also preserved 16/16 on those cursors, but block4096 drifted on later cursors 15/16 , 15/16 , and 6/16 . So block4096 is not a trusted restore format; block1024 is the current best compression/parity trade-off candidate. This is still not a production codec: the current scalar CPU codec is too slow for the critical path, and these gates cover one prompt/history, not the full prompt distribution. Build the Metal bridge once: make build/bridge.o Build the practical generation demo: crystal build --release --no-debug \ --link-flags="$ pwd /build/bridge.o -framework Metal -framework Foundation -lc++" \ bin/qwen35 generate.cr \ -o build/qwen35 generate Run greedy generation: ./build/qwen35 generate "The capital of France is" 64 Enable exact n-gram speculative decode for repeated text: QWEN35 NGRAM DECODE=1 ./build/qwen35 generate "The capital of France is" 64 Use the conservative automatic decode policy: QWEN35 DECODE POLICY=auto ./build/qwen35 generate "The capital of France is" 64 auto is the product-safe proposal-aware profile: it uses exact n-gram/cache proposals only when they are large enough to amortize verification, enables the candidate-shape risk gate plus a runtime corridor detector for untrusted suffix replay, and otherwise falls back to exact target decoding without invoking the neural draft model. Enable exact neural speculative decode with the Qwen 3.5 0.8B draft: QWEN35 SPECULATIVE DECODE=1 \ QWEN35 HEAD FULL ROWS GUARDED=1 \ ./build/qwen35 generate "The capital of France is" 64 Experimental same-weight proposal-route memory: crystal build --release --no-debug \ bin/qwen35 proposal route memory.cr \ -o build/qwen35 proposal route memory QWEN35 ROUTE CAL ROOT=/tmp/qwen35 route memory \ QWEN35 ROUTE CAL GEN=16 \ QWEN35 ROUTE CAL GAMMA=4 \ QWEN35 ROUTE CAL UPDOWN RANK=4 \ QWEN35 ROUTE CAL UPDOWN LAYERS=0,2,4 \ scripts/qwen35 proposal route calibrate.sh The route cache stores certified proposal-body choices such as baseline or pca updown for the GPU self-spec probe path. It is intentionally not a qwen35 generate decode-policy switch yet: the product generator currently has greedy, n-gram/cache, external-draft speculative, and GGUF-MTP paths, while the PCA-updown same-weight proposal body lives in bin/qwen35 deltanet fixed basis probe.cr . Use route memory to avoid repeating online route calibration in that probe corridor; do not count it as a product generation speedup until the same verifier/proposal corridor is wired into the normal generator. qwen35 generate can resolve the route table with QWEN35 SELF SPEC ROUTE MEMORY ROOT for diagnostics and future wiring, but it prints product self spec=unsupported and leaves the decode path unchanged. If QWEN35 SELF SPEC UPDOWN ADAPTERS PATH is also set, the generator validates the referenced PCA-updown adapter artifact against the selected route layers and rank through ML::GGUF::Qwen35SelfSpecPlan , then reports plan=pca updown|invalid adapter|baseline|route miss and adapter artifact=valid|invalid without executing the adapter. The PCA-updown FFN adapter data path is now shared in src/ml/gguf/qwen35 ffn updown adapter.cr : it owns the centered low-rank projection math, Hadamard/symmetric quant-dequant helpers, and the qwen35 ffn updown adapter v1 artifact dump/load format. The heavy self-spec scheduler is still probe-local. Manual route lookup/seed example: ./build/qwen35 proposal route memory \ --root /tmp/qwen35 route memory \ --prompt "def square x : return x x\n" \ --route-key code square \ --route pca updown \ --rank 4 \ --layers 0,2,4 Enable Qwen chat-template prompting and XML-style function calls: QWEN35 CHAT=1 \ QWEN35 CHAT SYSTEM="You are a tool-using assistant." \ QWEN35 TOOLS JSON=' {"type":"function","function":{"name":"get weather","description":"Get weather for a city","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required": "city" }}} ' \ ./build/qwen35 generate "Weather in Paris?" 64 QWEN35 TOOLS JSON uses the tool schema shape embedded in the Qwen tokenizer.chat template . Qwen 3.5/3.6 GGUFs use an XML-ish tool-call format, for example <tool call <function=... <parameter=... ... , not a constrained JSON-schema decoder. The CLI renders the model prompt with the Qwen chat tokens, uses the rendered prompt for tokenization and prompt-cache keys, and prints a parsed JSON summary when generated text contains Qwen <tool call blocks. For harnesses that expect normal JSON function-calling, keep the model-facing Qwen XML template and normalize only at the host boundary: QWEN35 CHAT=1 \ QWEN35 TOOL RESPONSE JSON=simple \ QWEN35 TOOLS JSON=' {"type":"function","function":{"name":"read file","description":"Read a file","parameters":{"type":"object","properties":{"path":{"type":"string"}},"required": "path" }}} ' \ ./build/qwen35 generate "Read src/foo.cr" 64 The emitted === Tool response JSON === block uses the CrystalBall-compatible shape. When QWEN35 TOOLS JSON is available, argument normalization is schema-aware for basic scalar types, so string fields stay strings while integer/number/boolean fields are emitted as typed JSON values: {"content":null,"tool calls": {"name":"read file","arguments":{"path":"src/foo.cr"}} } Set QWEN35 TOOL RESPONSE JSON=openai to emit OpenAI-style wrappers with id , type , and function.arguments as a JSON string. Multi-turn harnesses can pass OpenAI-style messages through QWEN35 MESSAGES JSON ; assistant tool calls are rendered back into Qwen XML blocks and tool messages are rendered as Qwen tool role messages. ../crystal ball can use this backend through its CogniQwen provider: crystal build --release bin/qwen35 generate -o build/qwen35 generate cd ../crystal ball CB COGNI QWEN=1 \ CB COGNI QWEN BIN=../cogni-ml/build/qwen35 generate \ crystal run src/cli.cr The lightweight adapter is useful for testing prompt/render and output parsing without loading the model: printf '<tool call \n<function=read file \n<parameter=path \nsrc/foo.cr\n</parameter \n</function \n</tool call \n' \ | crystal run bin/qwen35 tool json adapter.cr -- --parse-output --format=simple printf '{"messages": {"role":"user","content":"Read src/foo.cr"} ,"tools": }' \ | crystal run bin/qwen35 tool json adapter.cr -- --render-request Enable exact prompt cache: QWEN35 PROMPT CACHE=1 \ QWEN35 SESSION ID=demo \ ./build/qwen35 generate "The capital of France is" 64 With prompt cache enabled, qwen35 generate now caches two independent artifacts: - tokenized prompt text, so repeated prompts avoid the older external tokenizer process path; - exact prompt state, including optional full-prompt next-token metadata for long generations. The prompt-state lookup is longest-prefix based. If the new prompt extends a cached prompt, the CLI restores the verified prefix state and exact-replays only the uncached suffix before decode. A local product smoke on Qwen3.5 9B Q4 K M restored 12/14 prompt tokens, replayed 2 , skipped normal prompt prefill prefill ms=0.0 , and used cache route=prompt state restore . QWEN35 PROMPT CACHE FULL HIT MIN GEN controls when a full-prompt hit can use stored next-token metadata directly; the default favors longer generations so short requests do not merely shift first model work from prefill ms to decode ms . When QWEN35 PROMPT CACHE FAST FORWARD=1 and source-history cache is enabled, the CLI also writes a direct per-key output fast-forward certificate for fully generated spans. On a repeated same-session terminal request, this lets qwen35 generate validate model/session/prompt/output hashes and emit cached token ids/text before opening the GGUF. The older tokenized-prompt + source-history + manifest scan remains as a fail-closed fallback for legacy or tampered direct certificates. Fast-forward state artifacts can also be stored in the guarded compressed v2 format. Set QWEN35 PROMPT CACHE ARTIFACT CODEC=recurrent-bf16 to compress recurrent DeltaNet state while keeping KV rows raw; Metal restores these validated artifacts through the mmap encoded path. recurrent-int8 remains explicitly gated by QWEN35 PROMPT CACHE METAL INT8 RESTORE=1 . The native tokenizer is the default. Set QWEN35 NATIVE TOKENIZER OFF=1 only when comparing against the external llama-tokenize bootstrap path. Each generation ends with a compact request-phase summary: request summary: total ms=... model load ms=... draft load ms=... tokenize ms=... token cache hit=... state prepare ms=... source history lookup ms=... cache restore ms=... prefill ms=... decode ms=... source history save ms=... prompt tokens=... output tokens=... Use total ms for one-shot CLI latency. Use decode ms / output tokens only when comparing decoder loops after the model, tokenizer, prompt cache, and state setup costs are already accounted for. Measure warm resident-process request latency without changing product CLI semantics: crystal build --release --no-debug \ --link-flags="$ pwd /build/bridge.o -framework Metal -framework Foundation -framework MetalPerformanceShaders -lc++" \ bin/qwen35 warm request probe.cr \ -o build/qwen35 warm request probe ./build/qwen35 warm request probe \ --gen 16 \ --requests 5 \ --warmups 1 \ --quiet \ "The capital of France is" This probe loads model/tokenizer once, runs explicit warmup requests outside the measured set, then reports per-request total/tokenize/state-prepare/prefill/decode timings. Use it to evaluate daemon/server-mode economics; do not mix it with one-shot qwen35 generate totals. To isolate exact source-history replay economics without one-shot CLI cache, disk, or lazy Metal compile tax, run the resident replay mode: ./build/qwen35 warm request probe \ --source-replay \ --gen 64 \ --requests 3 \ --warmups 1 \ --quiet \ "alpha beta gamma delta alpha beta gamma delta" ./build/qwen35 warm request probe \ --source-replay \ --metal-profile \ --gen 64 \ --requests 1 \ --warmups 1 \ --quiet \ "alpha beta gamma delta alpha beta gamma delta" --source-replay seeds one exact generated span, restores the prompt state inside the same resident process, and verifies the known span in one chunk with the same tail-skip contract as source-history cache replay. It is a cache/session replay measurement, not plain generation throughput. To measure the real prompt-cache Store restore path in the same resident process, use --prompt-cache-replay . --resident-states=1 enables the hot in-memory restored-state template path: ./build/qwen35 warm request probe \ --prompt-cache-replay \ --resident-states 1 \ --gen 64 \ --requests 3 \ --warmups 1 \ --quiet \ "alpha beta gamma delta alpha beta gamma delta" To measure validated session-cache fast-forward, use --prompt-cache-fast-forward . This mode seeds an exact span once, stores the state after the processed part of that span, validates the full token-history hash, restores the cached state, and emits the cached output ids without running the verifier body: ./build/qwen35 warm request probe \ --prompt-cache-fast-forward \ --resident-states 1 \ --reuse-request-state \ --gen 64 \ --requests 3 \ --warmups 1 \ --quiet \ "alpha beta gamma delta alpha beta gamma delta" This is a cache-hit/session fast-forward measurement, not generation throughput. It is exact only when the cached span and state artifact are validated for the same model, tokenizer, prompt/session tokens, and runtime contract. --reuse-request-state models a daemon/server state pool: cached state restore overwrites the same prepared destination buffers each request instead of allocating a fresh destination state. On a local M2 Max Qwen3.5 9B Q4 K M smoke, resident fast-forward measured ~4.5 ms total for 64 cached tokens ~0.07 ms/tok before request-state pooling; a later 16-token BF16/live-KV smoke measured p50 total ~0.6-1.0 ms with request-state reuse. The same Store source replay path remains slower because it still runs the exact bulk verifier body. To measure the terminal direct-output certificate path inside a resident process, use --prompt-cache-direct-output . This mode seeds the same exact span and certificate once, then times only Store lookup output fast forward validation and cached id emission; it intentionally performs no state restore, prefill, or decoder work: ./build/qwen35 warm request probe \ --prompt-cache-direct-output \ --gen 16 \ --requests 7 \ --warmups 2 \ --quiet \ "The capital of France is" On a local M2 Max Qwen3.5 9B Q4 K M smoke, this measured p50 0.009 ms total for a 16-token cached span after warmup. The same build measured hot BF16/live-KV state fast-forward with --resident-states=1 and --reuse-request-state at p50 0.587 ms on the same prompt/settings. Treat direct-output certificate hits as the terminal repeated-output path; use state fast-forward only when the caller needs a continuation state after the cached span. To exercise the resident serving route order directly, use --prompt-cache-serving-route . Terminal requests try the direct output certificate first. If the caller requires continuation state after the cached span, add --serving-route-continuation ; this bypasses terminal id emission and uses the validated state fast-forward corridor instead: ./build/qwen35 warm request probe \ --prompt-cache-serving-route \ --gen 16 \ --requests 7 \ --warmups 2 \ --quiet \ "The capital of France is" ./build/qwen35 warm request probe \ --prompt-cache-serving-route \ --serving-route-continuation \ --resident-states 1 \ --reuse-request-state \ --artifact-codec recurrent-bf16 \ --live-kv-artifacts \ --gen 16 \ --requests 7 \ --warmups 2 \ --quiet \ "The capital of France is" The request summaries include route=direct output or route=state fast forward continuation . A local M2 Max smoke measured p50 0.010 ms for the terminal direct route and p50 0.661 ms for the continuation-state route on the same 16-token cached span. --serving-route-active-cursor is an upper-bound server-shape probe: it prewarms the continuation state once and then measures the already-owned active-session handoff, avoiding the reusable-cache copy that a shared Store must perform for isolation. For matrix gates, set QWEN35 MATRIX MODE=serving-route ; the TSV includes a routes column so policy changes are visible in apples-to-apples comparisons. For fail-closed diagnostics, --serving-route-direct-miss omits the direct output certificate while keeping the exact-known-span artifact. Terminal requests then fall back to route=source history direct output fallback without restoring state; requests that need continuation state still use the validated state corridor. The route decision is also available to product code as ML::GGUF::Qwen35ServingRoute.serve exact cached span in src/ml/gguf/qwen35 serving route.cr . For resident servers, ML::GGUF::Qwen35ResidentSession in src/ml/gguf/qwen35 resident session.cr owns route counters and an optional active continuation cursor while keeping shared Store restores copy-safe. For a repeatable raw-vs-compressed cache-artifact gate, use the matrix runner: scripts/qwen35 cache artifact matrix.sh By default it builds/uses /tmp/qwen35 warm request probe matrix , runs five prompt classes through --prompt-cache-fast-forward , and reports tab-separated rows with mode , artifact codec, live-KV policy, request totals, and phase timings. Useful overrides: QWEN35 MATRIX PROMPT LIMIT=1 \ QWEN35 MATRIX REQUESTS=1 \ QWEN35 MATRIX WARMUPS=0 \ QWEN35 MATRIX CODECS="raw recurrent-bf16" \ QWEN35 MATRIX LIVE KV="0 1" \ QWEN35 MATRIX REUSE REQUEST STATE=1 \ scripts/qwen35 cache artifact matrix.sh QWEN35 MATRIX MODE=direct-output \ QWEN35 MATRIX REQUESTS=7 \ QWEN35 MATRIX WARMUPS=2 \ scripts/qwen35 cache artifact matrix.sh QWEN35 MATRIX MODE=direct-output collapses the artifact axes to codec=direct and live kv=na , because this mode measures only the resident direct output-certificate validator and cached id emission. Use the matrix for cache-artifact economics only. It does not measure first-run prefill or plain generation throughput. For prefill/decode profiling logs, use the graph-atlas helper to turn Qwen35Metal.Profile reports into ranked LTP/WBA windows: QWEN35 PREFILL PHASE PROFILE=1 /tmp/qwen35 prefill attribution \ --prompt=64 --warmup=1 --reps=1 --prepare-state \ --load-warning-threshold=0 --load-total-warning-threshold=0 \ /tmp/qwen35 prefill profile.log scripts/qwen35 profile atlas.cr /tmp/qwen35 prefill profile.log --top=8 The atlas is an attribution aid, not a benchmark by itself. Phase profiling intentionally changes command-buffer boundaries, so use paired wall timing before promoting any kernel or scheduling change. Useful Qwen environment switches: | Variable | Effect | |---|---| QWEN35 PROMPT CACHE=1 | Enable exact prompt-state cache lookup/save in qwen35 generate . | QWEN35 PROMPT CACHE ROOT=/path | Override prompt-cache artifact root. | QWEN35 PROMPT TOKEN CACHE OFF=1 | Disable tokenized-prompt cache lookup/save while keeping prompt-state cache enabled. | QWEN35 PROMPT CACHE FAST FORWARD=1 | With prompt cache and source-history enabled, save and use validated post-span state artifacts so exact session-cache hits can emit cached spans without verifier recompute. Default off; falls back when source prefix, token hash, or artifact validation fails. | QWEN35 PROMPT CACHE PREWEIGHT FAST FORWARD OFF=1 | Diagnostic switch: disable pre-weight terminal fast-forward exits so qwen35 generate can exercise the post-weight serving-route/state-restore path. Default off; normal product routing still tries zero-GGUF/zero-weight terminal hits first. | QWEN35 PROMPT CACHE ARTIFACT CODEC=recurrent-bf16 | Store validated fast-forward state artifacts in compressed v2 BF16 recurrent format. KV rows remain raw. Default is raw; recurrent-int8 is available only with the explicit Metal INT8 gate. | QWEN35 PROMPT CACHE ARTIFACT CODEC BLOCK=8 | Block size for recurrent-int8 prompt-cache artifacts. Ignored for raw/BF16. | QWEN35 PROMPT CACHE LIVE KV ARTIFACTS=1 | Store prompt-cache artifacts in v3 live-KV format: full recurrent state plus only live full-attention KV rows. This is opt-in while broader validation continues. | QWEN35 PROMPT CACHE METAL INT8 RESTORE=1 | Explicitly allow Metal restore of validated recurrent-int8 artifacts. Default off because INT8 remains approximate and needs stronger prompt/session validation than BF16. | QWEN35 PROMPT CACHE FULL HIT MIN GEN=64 | Minimum requested generation length before a full-prompt cache hit can skip suffix replay and use stored next-token metadata. Lower values are useful for experiments but can move first model work into decode ms without improving total wall time. | QWEN35 PROMPT CACHE RESIDENT STATES=0 | Number of restored prompt-cache states to keep hot in a resident process. 0 disables the in-memory state cache. Positive values avoid rereading and redecompressing .qkv artifacts on repeated same-process session hits. | QWEN35 SELF SPEC ROUTE MEMORY ROOT=/path | Diagnostic/future-wiring hook: resolve a same-weight self-spec proposal route in qwen35 generate using the shared route table. Current product generation does not execute the PCA-updown self-spec runner, so hits are reported and the decode path remains unchanged. | QWEN35 SELF SPEC ROUTE KEY=KEY | Optional caller-certified key for QWEN35 SELF SPEC ROUTE MEMORY ROOT ; without it, lookup uses exact prompt text and token ids. | QWEN35 SELF SPEC UPDOWN ADAPTERS PATH=/path.json | Optional diagnostic/future-wiring hook paired with QWEN35 SELF SPEC ROUTE MEMORY ROOT : validate a qwen35 ffn updown adapter v1 artifact for a selected pca updown route and report validity without changing decode behavior. | QWEN35 NATIVE TOKENIZER OFF=1 | Disable the native Crystal Qwen BPE encoder and use the external llama-tokenize bootstrap path. | QWEN35 PREPARE STATE OFF=1 | Disable eager Metal state-buffer preparation in qwen35 generate . By default the CLI prepares KV/DeltaNet buffers before timing prompt ingest. | QWEN35 METAL PROFILE=1 | Enable Metal dispatch/profile attribution for the timed decode region in qwen35 generate . The report includes wave/group timings, prefill/source-replay phase traces, matmul logical traffic, conversion traffic, and CPU fallback counts. | QWEN35 CHAT=1 | Render the input through the minimal Qwen 3.5/3.6 chat-template path before tokenization. | QWEN35 CHAT SYSTEM="..." | Optional system message used by the chat-template renderer. | QWEN35 TOOLS JSON=' ... ' | Enable Qwen XML-style function-calling prompt rendering and parsed <tool call output reporting. The value must be a JSON array of tool definitions. | QWEN35 MESSAGES JSON=' ... ' | Render an OpenAI/CrystalBall-style message array through the Qwen chat-template path. Supports user , system , assistant with tool calls , and tool messages. | QWEN35 TOOL RESPONSE JSON=simple|openai | Emit a machine-readable === Tool response JSON === block after generation. simple matches CrystalBall's local provider shape; openai wraps calls as OpenAI-style function tool calls. With QWEN35 TOOLS JSON , scalar arguments are normalized using the tool schema where possible. | QWEN35 CONSTRAINED LITERAL PREFIX='...' | Experimental greedy-only constrained decoding probe. Forces an exact literal prefix using tokenizer-derived allowed-token frontiers and the constrained Q6 head path, then falls back to normal greedy decode after the literal completes. Currently incompatible with prompt-cache/speculative/n-gram fast paths. | QWEN35 CONSTRAINED TOOL CALL PREFIX=1 | Experimental greedy-only structured tool-call mode. Requires QWEN35 TOOLS JSON ; constrains Qwen XML <tool call \n<function=... \n prefixes over the available function names, required <parameter=... \n tags in schema order, lets the model choose schema-valid optional parameter tags before final close, constrains finite enum/boolean values and small bounded integer ranges, falls back for free-form single-line values, constrains the inter-parameter and final closing tags after value newlines, stops generation after a complete constrained tool call, batches deterministic literal spans by default, and emits a tool constraint summary line with constrained-stage, forced-span, and free-form fallback counts. | QWEN35 CONSTRAINED FORCE SPAN OFF=1 | Disable deterministic literal-span batching inside constrained structured decoding. This is a diagnostic kill switch; the default batches grammar-proven spans with exact body-only state updates and skips intermediate lm-head ranking. | QWEN35 DECODE POLICY=greedy|ngram|speculative|mtp|auto | Explicit decode-mode selector. auto chooses the exact fail-closed n-gram path with risk gating; explicit policy overrides legacy mode envs. mtp requires QWEN35 MTP GGUF PATH and remains explicit/default-off. | QWEN35 TRACE STEPS OFF=1 | Suppress per-token/per-cycle trace lines in qwen35 generate while keeping summaries and final output. | QWEN35 QUIET=1 | Alias for suppressing per-step traces in qwen35 generate ; useful for cleaner local timing. | QWEN35 NGRAM DECODE=1 | Enable exact n-gram speculative decode in qwen35 generate . | QWEN35 NGRAM GAMMA=32 | Maximum n-gram verifier chunk size. | QWEN35 NGRAM MIN=6 | Minimum repeated suffix length before n-gram drafting. | QWEN35 NGRAM MAX=8 | Maximum suffix length to search for n-gram drafting. | QWEN35 NGRAM MIN CANDIDATES=N | Skip n-gram proposals shorter than N candidates. In auto , the default is 8 ; explicit ngram keeps the old default 0 unless set. | QWEN35 NGRAM STAGE MIN=N | Split only n-gram verifier chunks with at least N candidates into staged subchunks. In auto , the default is QWEN35 NGRAM GAMMA + 1 , so the common full chunk is kept intact unless overridden. Explicit ngram keeps the old default 0 unless set. | QWEN35 NGRAM RISK GATE=0|1 | In auto , the exact candidate-shape risk gate is enabled by default; set 0 to disable. Explicit ngram keeps the old default off unless set to 1 . | QWEN35 NGRAM RISK MIN SIZE=16 | Candidate size threshold used by the n-gram risk gate. This is independent from QWEN35 NGRAM STAGE MIN , so staging can be disabled without weakening fail-closed risk checks. | QWEN35 NGRAM CORRIDOR GATE=0|1 | In auto , require runtime repeat-corridor evidence for untrusted local suffix n-gram proposals. Trusted source-history cursor replay bypasses this gate after prefix validation. | QWEN35 NGRAM CORRIDOR MATCH LEN MIN=8 | In auto , allow untrusted suffix proposals when the repeated suffix match reaches this length. Explicit ngram defaults to 0 . | QWEN35 NGRAM CORRIDOR LAG8 MIN=2.0 | In auto , lag-8 repetition alone is disabled as a product certificate after template false positives; set a lower value only for A/B. Explicit ngram defaults to 0.5 . | QWEN35 NGRAM RECURSIVE OFF=1 | Disable recursive n-gram extension through scratch history. | QWEN35 NGRAM DISABLE AFTER REJECT OFF=1 | Exploration mode: keep trying n-gram chunks after first rejection. | QWEN35 NGRAM REPLAY ON REJECT=1 | Research/fast-path mode: skip n-gram target-state backups and rebuild the exact target state only after a non-final n-gram reject. Use with the default auto risk gate; it can regress badly when a large bad n-gram chunk is forced through verification. | QWEN35 NGRAM CACHE MIN REMAINING=64 | In auto , require at least this many remaining requested tokens before using source-history/cache n-gram proposals. This keeps short cache-hit requests on the cheaper exact replay/greedy path. Explicit ngram defaults to 0 unless set. | QWEN35 SPECULATIVE DECODE=1 | Enable exact neural speculative decode in qwen35 generate using the 0.8B draft. | QWEN35 DRAFT MODEL=/path | Override the Qwen 3.5 draft GGUF used by neural speculative decode. | QWEN35 SPEC GAMMA=4 | Initial neural draft chunk size in qwen35 generate . | QWEN35 SPEC MAX GAMMA=32 | Maximum adaptive neural draft chunk size. | QWEN35 SPEC PLAIN FALLBACK OFF=1 | Disable target-only fallback after low-gamma speculative rejection. Useful for A/B experiments; default fallback is faster on rejection-heavy prompts. | QWEN35 SPEC PLAIN FALLBACK GAMMA=2 | Gamma threshold at or below which rejected neural speculative decode falls back to target-only generation. | QWEN35 SPEC BOOTSTRAP GAMMA=N | Default-off neural speculative jump after a fully accepted initial chunk. Can help 100%-accept runs; may regress prompts that reject after an accepted prefix. | QWEN35 SPEC SINGLE FAST OFF=1 | Disable the exact gamma=1 accepted-token fast path in neural speculative decode. Mostly useful when target-only fallback is disabled for A/B experiments. | QWEN35 SPEC VERIFY=chunk-inplace|hybrid|serial | Choose neural speculative verifier strategy. Default chunk-inplace is best for high-accept prompts; hybrid can help first-cycle partial-reject prompts. | QWEN35 SPEC SKIP DRAFT BEFORE FALLBACK OFF=1 | Disable the exact optimization that skips draft resync work when a rejection is guaranteed to enter target-only fallback. | QWEN35 SPEC SKIP DRAFT BACKUP BEFORE FALLBACK OFF=1 | Disable the matching draft-backup skip before fallback-bound speculative chunks. | QWEN35 HEAD FULL ROWS GUARDED=1 | Experimental speculative-verifier accelerator for large accepted chunks; uses a margin guard and exact fallback for low-margin rows. | QWEN35 HEAD FULL ROWS MARGIN=0.25 | Margin threshold for the guarded full-row verifier route. Higher is safer but falls back more often. | QWEN35 FFN DOWN ADD FUSED OFF=1 | Disable decode-wave FFN-down residual-add fusion for Q4/Q6 target and Q8 draft experiments. | QWEN35 Q4K PAIR H16 MIN BATCH=64 | Tune the prefill Q4 gate/up shared H16 conversion threshold. The current default enables sharing from pp64 upward after refreshed A/B showed a small exact win. | QWEN35 Q4K H16 B48 OFF=1 | Disable the exact 48-token Q4 K H16 prefill tile. The default route uses this only for exact 48-token chunks. | QWEN35 Q4K H16 B64 OFF=1 | Disable the exact wide-batch Q4 K H16 prefill GEMM. The default route uses a 64-token batch tile for prompt chunks that are exact multiples of 64; irregular chunk sizes stay on the older 32-token tile to avoid tail regressions. | QWEN35 Q4K H16 B64 TAIL MIN=N | Experimental exact prefill probe: for non-64 prompt chunks at least N tokens, allow the B64 Q4 K H16 tile and its tail-safe B64 up+SwiGLU fusion on the underfilled final tile. Exact row-pack shapes with dedicated tiles stay on their proven routes. Default off because prior long-prompt tail evidence is shape-sensitive; use for A/B only. | QWEN35 Q4K H16 B80 OFF=1 | Disable the exact 80-token Q4 K H16 prefill tile. The default route uses this only for exact 80-token chunks. | QWEN35 Q4K H16 B96 OFF=1 | Disable the exact 96-token Q4 K H16 prefill tile. The default route uses this only for exact 96-token chunks; wider irregular chunks still avoid B96 after pp160/pp192 regressions. | QWEN35 Q4K H16 B112 OFF=1 | Disable the exact 112-token Q4 K H16 prefill tile. The default route uses this only for exact 112-token chunks. | QWEN35 ADDNORM H16 FFN=1 | Experimental exact prefill route: residual add+RMSNorm writes H16 rows directly for following Q4 H16 FFN gate/up GEMMs. Default off; current evidence is mixed and it is retained only as an attribution/probe knob. | QWEN35 SWIGLU H16 DOWN=1 | Experimental exact prefill route: SwiGLU writes H16 activations directly for FFN-down Q4/Q5/Q6 batch GEMM consumers. Default off; it removes large conversion traffic at long prompts, but paired wall timing is mixed and not a promotion signal yet. | QWEN35 RMSNORM H16 PROJ=1 | Experimental exact prefill route: RMSNorm writes f32 plus H16 rows so Q4/Q5/Q6 projection GEMMs can skip their separate input conversion. Default off; useful for attribution, but current paired wall timing is noise-level. | QWEN35 DN POST H16 OPROJ=1 | Experimental exact prefill route: recurrent DeltaNet post-gate writes H16 rows directly for following o proj GEMMs. Default off; completes the conversion-atlas probe set, but current wall timing is neutral. | QWEN35 REC PROJ SHARED H16 OFF=1 | Disable the exact recurrent prefill projection optimization that shares one H16 input conversion between Q5 qkv and Q4 gate GEMMs. | QWEN35 PREFILL CHUNK OFF=1 | Force older non-chunked prefill path. | QWEN35 DECODE WAVE OFF=1 | Force older non-wave decode path. | Structured tool-call span batching can be regression-tested with: REPS=2 scripts/qwen35 structured span suite.sh The current Qwen API is low-level and intended for native inference experiments: require "ml/gguf/qwen35 cpu" require "ml/gguf/qwen35 weights" model = "/path/to/Qwen3.5-9B-Q4 K M.gguf" weights = ML::GGUF::Qwen35Weights.from gguf model state = ML::GGUF::Qwen35CPU::State.new weights.hparams, max seq: 1024 prompt ids = 760 i32, 6511 i32, 314 i32, 9338 i32, 13 i32 next id, next logit = ML::GGUF::Qwen35CPU.prefill tokens top1 weights, prompt ids, 0, state 64.times do |i| puts next id next id, next logit = ML::GGUF::Qwen35CPU.forward top1 weights, next id, prompt ids.size + i, state end When linking an executable that uses Metal, include the bridge object and Apple frameworks: crystal build your app.cr \ --link-flags="$ pwd /build/bridge.o -framework Metal -framework Foundation -lc++" For CPU-only builds: crystal build -Dcpu only your app.cr Build the matched benchmark: crystal build --release --no-debug \ --link-flags="$ pwd /build/bridge.o -framework Metal -framework Foundation -lc++" \ bin/benchmark qwen vs llama.cr \ -o build/benchmark qwen vs llama Run a normal first-run prefill/decode comparison: ./build/benchmark qwen vs llama \ --model ~/.cache/lm-studio/models/lmstudio-community/Qwen3.5-9B-GGUF/Qwen3.5-9B-Q4 K M.gguf \ --llama-bench ~/SrcArchives/AI/llama.cpp/build/bin/llama-bench \ --prompt=64 \ --gen=64 \ --reps=5 \ --warmup=2 For publishable measurements, wait for a quiet host: ./build/benchmark qwen vs llama \ --prompt=64 \ --gen=64 \ --reps=5 \ --warmup=2 \ --wait-quiet-ms=60000 \ --require-quiet Additional benchmark modes: Default native decode now matches llama-bench tg : decoder body only, with no output logits/head readback. Product-shaped greedy decode is: ./build/benchmark qwen vs llama --native-decode-top1 Fresh State per repetition, but Metal state buffers are prepared before the timed prefill. This measures prompt ingest without first-touch buffer allocation/zeroing in the timed region. ./build/benchmark qwen vs llama --native-prefill-prepare-state State buffers allocated once, then reset between reps. ./build/benchmark qwen vs llama --native-prefill-prealloc Exact prompt-cache restore after one seeded native prefill. ./build/benchmark qwen vs llama --native-prefill-cache Prompt-cache prefix restore plus exact replay of the last N prompt tokens. ./build/benchmark qwen vs llama --native-prefill-cache-prefix-suffix=8 Latest guarded relaxed-host Qwen3.5-9B Q4 K M body-only rows on M2 Max: | prompt/gen | cogni-ml pp | llama.cpp pp | pp gap | cogni-ml tg | llama.cpp tg | tg gap | |---|---|---|---|---|---|---| | 64/64 | 480.06 tok/s | 458.85 tok/s | +4.62% | 53.75 tok/s | 48.15 tok/s | +11.63% | | 256/64 | 567.60 tok/s | 566.21 tok/s | +0.25% | 53.43 tok/s | 48.26 tok/s | +10.72% | | 1024/64 | 582.74 tok/s | 574.77 tok/s | +1.39% | 53.14 tok/s | 48.24 tok/s | +10.16% | These rows use --native-prefill-prealloc , --threads=8 , and disabled load warnings. Treat them as guarded relaxed measurements, not publishable quiet-host ABBA evidence. For Qwen3.6 MTP / quant baselines, first inspect the local/HF matrix: crystal build --release bin/qwen36 mtp baseline matrix.cr \ -o build/qwen36 mtp baseline matrix ./build/qwen36 mtp baseline matrix The matrix prints local-path detection plus llama.cpp command lines for the current plain Q4 K M target and external MTP GGUF baselines such as unsloth/Qwen3.6-27B-MTP-GGUF:IQ4 NL , unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q4 K XL , and AtomicChat/Qwen3.6-27B-UDT-MTP-GGUF:Q4 K XL MTP . Use --run-available only after the relevant files are local and the llama.cpp binary advertises draft-mtp . benchmark qwen vs llama also accepts llama.cpp KV-cache options for matching long-context/product configurations: ./build/benchmark qwen vs llama \ --model ~/.cache/lm-studio/models/lmstudio-community/Qwen3.6-27B-GGUF/Qwen3.6-27B-Q4 K M.gguf \ --prompt=512 \ --gen=128 \ --llama-cache-k=q8 0 \ --llama-cache-v=q4 0 Boundary: native cogni-ml currently supports the K-quants used by our Q4 K M/Q5 K/Q6 K/Q8 0 paths. IQ/UD MTP GGUFs are external llama.cpp baselines until native IQ/UD quant loaders are implemented. Fresh local M2 Max 64GB relaxed-load snapshot after the shared-H16 recurrent projection cleanup, Qwen 3.5 9B Q4 K M, llama.cpp llama-bench , prompt=64 , gen=64 , reps=3 , warmup=1 , flash-attention off: | Mode | cogni-ml | llama.cpp | Gap | |---|---|---|---| | First-run prefill | 426.70 tok/s p50 | 455.10 tok/s avg | -6.24% | | Fresh state, prepared Metal buffers | 449.73 tok/s p50 | 464.78 tok/s avg | -3.24% | | Prefill with preallocated state | 448.60 tok/s p50 | 465.91 tok/s avg | -3.71% | | Prompt-cache restore | 1350.65 tok/s p50 | 465.80 tok/s avg | +189.97% | | Plain greedy decode, first-run bench | 48.67 tok/s p50 | 46.67 tok/s avg | +4.29% | | Plain greedy decode, prepared-state bench | 48.59 tok/s p50 | 46.43 tok/s avg | +4.67% | | Plain greedy decode, preallocated bench | 48.51 tok/s p50 | 46.63 tok/s avg | +4.05% | | Plain greedy decode, prompt-cache bench | 48.52 tok/s p50 | 46.58 tok/s avg | +4.18% | Notes: - The table is a local engineering snapshot, not a lab-clean public benchmark. - First-run prefill is still behind llama.cpp on this machine. The native wins currently come from state reuse, prompt-cache restore, and exact speculative decode. --native-prefill-prepare-state uses a fresh State per repetition but calls Qwen35CPU.prepare state metal before timing. This is useful for server-style latency where a session object can be prepared before the prompt arrives. --native-prefill-cache measures exact restore of a previously computed prompt state; it is not a first-run prefill replacement.- Short decode runs are noisy on a desktop system. The two plain decode rows above are intentionally both shown: treat plain decode as parity-to-faster, not as a stable public margin without a quiet rerun. Same-host CUDA snapshot, RTX 5060 Ti, Qwen 3.5 9B Q4 K M, gen=64 : | Mode | Speed | Notes | |---|---|---| | llama.cpp CUDA plain decode | 67.18 tok/s, 14.89 ms/tok | llama-bench tg64 on the same host/model | | cogni-ml CUDA plain greedy probe | ~42.6-42.9 tok/s, ~23.3-23.5 ms/tok | full resident CUDA target path, no proposal reuse | | cogni-ml CUDA source/cache cursor, risk-gated | 71.76 tok/s, 13.94 ms/tok | exact output; 62/62 active-verified tokens plus two serial cursor advances | | cogni-ml CUDA trusted source/cache cursor | 86.61 tok/s, 11.55 ms/tok | exact output; 64/64 active-verified tokens, chunks 4,4,8,16,16,16 | | cogni-ml CUDA trusted bulk replay | 91.69 tok/s, 10.91 ms/tok | exact output; 64/64 active-verified tokens, one 64 -token WBA chunk | | cogni-ml CUDA trusted artifact restore, device-resident | restore-only lower bound: 0.189 ms / 64 cached tokens | no verifier recompute; requires validated post-span state artifact | | cogni-ml CUDA trusted artifact restore, host-backed | 12.259 ms / 64 cached tokens | restores a full max seq=128 decode state from host memory; excludes durable IO/hash lookup | | cogni-ml CUDA trusted artifact restore, host-backed live KV | 11.358 ms / 64 cached tokens | restores recurrent state plus live KV rows only; recurrent state dominates at short context | | cogni-ml CUDA trusted artifact IO/hash, live KV | write/read/hash/restore: 17.695/42.046/59.556/13.003 ms | persistent-path diagnostic for a 56.9 MB artifact; exact cached output still 64/64 | | cogni-ml CUDA recurrent block-INT8 artifact codec | recurrent bytes 52.7MB - ~13.2MB ; one-token continuation parity passed on tested cursors | scalar CPU encode/decode is too slow; treat as codec feasibility evidence, not production restore | | invalid trusted cursor | ~42.3 tok/s, ~23.64 ms/tok | fails closed: zero proposals, active verifier disabled, near plain fallback | CUDA cache-replay caveats: - The fast rows are not arbitrary generation. They measure exact replay from a validated session/source cursor, which is the intended primitive for prompt/session cache hits and repeated known spans. - Exact greedy parity is preserved by verifying every proposed token through the target stack before committing it. - Wrong cursors must fail closed. The trusted-source mode is only trusted after the source-prefix gate passes; otherwise it falls back near plain decode speed instead of accepting proposal tokens. - The host-backed live-KV row excludes durable cache lookup, artifact IO, and hashing; the IO/hash row includes a simple persistent-path file write/read plus host SHA-256 tool timing, but still excludes pg/session lookup and production artifact indexing. - Trusted artifact restore is a cache-hit fast-forward path, not speculative decoding. It is only exact if the restored state artifact and emitted token span are validated against the same model/tokenizer/config/source contract. Local Metal resident cache snapshot, M2 Max, Qwen 3.5 9B Q4 K M, prompt alpha beta gamma delta alpha beta gamma delta : | Mode | Speed | Notes | |---|---|---| | cogni-ml Metal plain greedy resident | ~24.97 ms/tok | fresh request state, same process, no proposal reuse | | cogni-ml Metal exact source replay | ~4.60 ms/tok | synthetic resident prompt-state copy plus exact bulk verifier | | cogni-ml Metal Store source replay, resident states off | ~5.82 ms/tok | real prompt-cache Store path; cold artifact restore dominates restore phase | | cogni-ml Metal Store source replay, resident states on | ~4.51-4.55 ms/tok | real Store hot state path; remaining wall is verifier body | | cogni-ml Metal Store fast-forward, resident states off | ~1.45 ms/tok for 64 cached tokens | cold artifact restore; no verifier body | | cogni-ml Metal Store fast-forward, resident states on | ~0.07 ms/tok for 64 cached tokens; ~0.02 ms/tok for 256 cached tokens | hot validated state restore plus cached token emission; no verifier body | qwen35 generate direct output fast-forward | ~1-2 ms total for a 16-token cached span even with 5k irrelevant legacy manifest rows per cache file | one-shot CLI hit reads a per-key output certificate, validates prompt/output/text/exact-span hashes before opening GGUF, emits cached ids/text, and exits | | Metal encoded BF16 artifact read+restore | ~26.35 ms for a 28.4MB BF16 recurrent artifact in a one-prompt smoke | direct encoded BF16 recurrent decode into prepared Metal buffers; avoids CPU BF16 decode path measured at ~715 ms , but resident decoded templates remain faster at ~1.7 ms restore-only | Latest focused resident-cache smoke, M2 Max, Qwen3.5 9B Q4 K M, prompt14/gen8 /tmp/qwen35 warm cache modes 20260528095435 : plain resident greedy p50 265.185 ms ; Store replay without resident state p50 156.728 ms with p50 restore 44.817 ms ; Store replay with --resident-states 2 p50 115.932 ms with p50 restore 3.367 ms ; state fast-forward p50 3.657 ms ; serving-route continuation p50 4.304 ms ; active cursor p50 0.007 ms ; direct-output certificate p50 0.015 ms . These are warm-process session-cache numbers, not one-shot CLI latency. Metal cache caveat: source replay and fast-forward are different contracts. Source replay still verifies the known span through the exact target stack. Fast-forward skips that body and is only legal after validating the cached post-span state artifact and full emitted-token history. The CLI output-only hit also requires cached generated text for exactly the requested output length; it is exact for the emitted cached text, but it does not restore a continuation state because the process exits. Neural draft harness with Qwen 3.5 0.8B Q8 0: crystal build --release --no-debug \ --link-flags="$ pwd /build/bridge.o -framework Metal -framework Foundation -lc++" \ bin/qwen35 speculative accept.cr \ -o build/qwen35 speculative accept ./build/qwen35 speculative accept \ --target ~/.cache/lm-studio/models/lmstudio-community/Qwen3.5-9B-GGUF/Qwen3.5-9B-Q4 K M.gguf \ --draft ~/.cache/lm-studio/models/lmstudio-community/Qwen3.5-0.8B-GGUF/Qwen3.5-0.8B-Q8 0.gguf \ --tokens 64 \ --ngram \ "The capital of France is" Cheap-proposal-only policy for repeated/template spans: QWEN35 SPEC NGRAM MIN CANDIDATES=8 \ ./build/qwen35 speculative accept \ --target ~/.cache/lm-studio/models/lmstudio-community/Qwen3.6-27B-GGUF/Qwen3.6-27B-Q4 K M.gguf \ --draft ~/.cache/lm-studio/models/lmstudio-community/Qwen3.5-0.8B-GGUF/Qwen3.5-0.8B-Q8 0.gguf \ --tokens 32 \ --ngram \ --ngram-risk-gate \ --ngram-target-only \ "alpha beta gamma alpha beta gamma alpha beta gamma alpha" Target-only n-gram speculative harness: crystal build --release --no-debug \ --link-flags="$ pwd /build/bridge.o -framework Metal -framework Foundation -lc++" \ bin/qwen35 ngram speculative.cr \ -o build/qwen35 ngram speculative ./build/qwen35 ngram speculative \ --tokens 64 \ --gamma 32 \ --min-ngram 6 \ "The capital of France is" Both harnesses replay/check exact greedy target output by default unless their CLI explicitly says otherwise. Fresh local speculative smoke, same M2 Max 64GB and Qwen 3.5 9B target: | Mode / prompt | Effective speed | Plain target | Notes | |---|---|---|---| Neural draft, The capital of France is | 15.38 ms/tok, 65.01 tok/s | 21.98 ms/tok, 45.49 tok/s | 100% accepted, 64/64 candidates | Neural draft, def fibonacci n : | 21.06 ms/tok, 47.48 tok/s | 21.71 ms/tok, 46.07 tok/s | falls back after rejection; small but safe win | N-gram + neural, The capital of France is | 10.10 ms/tok, 98.98 tok/s | 21.91 ms/tok, 45.64 tok/s | repeated-text path, 48/48 n-gram candidates accepted | Experimental guarded full-row verifier + neural, The capital of France is | 14.32 ms/tok, 69.82 tok/s | 22.36 ms/tok, 44.73 tok/s | QWEN35 HEAD FULL ROWS GUARDED=1 , 0 fallback rows in this run | Experimental guarded full-row verifier + n-gram + neural, The capital of France is | 9.20 ms/tok, 108.64 tok/s | noisy target run | QWEN35 HEAD FULL ROWS GUARDED=1 , 48/48 n-gram candidates accepted | Speculative decode caveats: - The speculative paths are exact greedy verification paths, not approximate sampling shortcuts. - Neural speculative speed depends on draft acceptance. High-accept prompts are faster; rejection-heavy prompts quickly fall back to plain target decode. - In qwen35 generate , neural speculative decode is useful for longer high-accept generations. In a local 64-token smoke, The capital of France is measured 20.40 ms/tok greedy, 16.61 ms/tok neural speculative, and 15.10 ms/tok neural speculative with guarded full-row verification. A 32-token smoke was slower due fixed draft/verifier overhead. - N-gram speculation is a workload-specialized path for repeated/generated-template text. QWEN35 DECODE POLICY=auto is the recommended product profile: risk-gated, runtime-corridor gated for untrusted local suffix replay, min candidates=8 , no neural fallback, and exact target-only fallback outside clean cheap-copy spans. - Prompt cache and source-history n-gram are request-level accelerators, so judge them by total ms , not decode-only ms/tok . In a local repeated long-generation smoke, native tokenization removed the external tokenizer cost ~560 ms - ~0.1 ms , tokenized-prompt cache made repeated tokenization effectively 0.0 ms , and full-prompt cache plus source-history n-gram accepted 128/128 generated tokens with total wall around 1.55 s . For short generations, the default QWEN35 PROMPT CACHE FULL HIT MIN GEN=64 avoids a misleading path that only shifts first model work between cache restore ms and decode ms . - In the research acceptance harness, --ngram-target-only / QWEN35 SPEC NGRAM TARGET ONLY=1 skips neural draft fallback after the cheap n-gram proposal source and uses exact target-only steps instead. On a 27B mixed JSONL gate it beat neural default on 5/7 prompts with paired ratio 0.909x when combined with --ngram-risk-gate , but its average speed was near plain target decode; the real win is on clean repeated spans. - The n-gram risk gate now also catches small-period prefix overruns such as IP/YAML tails. A focused 27B probe kept clean alpha beta gamma repeats at ~2.58x while turning a YAML overrun from 0.80x into fail-closed target-only 1.006x . - The productization smoke after the structured-tail fix kept clean 27B repeats fast ~2.55x over plain , made a YAML-like overrun fail closed 1.003x , and had ngram target only risk beat neural default on the tiny paired suite 0.815x ratio . Treat this as opt-in CLI evidence, not a broad default claim. QWEN35 NGRAM REPLAY ON REJECT=1 is exact but deliberately opt-in. It removes rollback-copy overhead on high-confidence accepted n-gram chunks; on the local 27B+0.8B repeat8 harness it improved ngram router16 risk from 30.85 to 30.21 ms/tok . A forced no-risk YAML reject regressed from 71.53 to 95.86 ms/tok , so this is not a broad default.- N-gram verifier chunks temporarily disable guarded full-row verification even if QWEN35 HEAD FULL ROWS GUARDED=1 , because partial n-gram rejection exposed a close-row guard failure during adversarial CLI testing. QWEN35 HEAD FULL ROWS GUARDED=1 is still an experimental research switch. The harness checks final output against plain greedy target output, but the route is not broad-defaulted because it relies on a full-row F16 top1 margin guard.- These numbers are effective decode throughput after prompt prefill; they do not make first-run prefill faster. The embedding path targets nomic-embed-text-v2-moe with a fully native Metal compute pipeline. require "ml" require "ml/gguf/nomic bert" require "ml/gguf/metal backend" require "ml/metal/compute graph" ML::Metal::Device.init model = ML::GGUF::NomicBertMoE.from gguf "path/to/model.gguf", ML::GGUF::MetalBackend.new embedding = model.embed "Your text here" Apple M2 Max, 38 GPU cores: | Tokens | Latency | |---|---| | 20 | 14 ms | | 94 | 16 ms | | 196 | 33 ms | | 433 | 70 ms | - simdgroup-matrix GEMM for Q5 K/Q6 K dequant+multiply. - Batched expert GEMM for MoE experts. - ComputeGraph wave scheduling with offset-aware dependency analysis. - Fused QKV split/RoPE, gate/softmax/top-k, scatter, and norm kernels. - GPU-driven dispatch where useful. | Model | Format | Status | |---|---|---| Qwen3.5-9B | GGUF Q4 K M | Native Metal text generation path, active optimization target. | Qwen3.5-0.8B | GGUF Q8 0 | Native draft model path for speculative decode harnesses. | Qwen3.6-27B | GGUF Q4 K M target | Planned/experimental scale-up target. | nomic-embed-text-v2-moe | GGUF Q5 K M | Native Metal embedding pipeline. | | BERT-like encoders | GGUF | Via NomicBertMoE when the architecture matches. | | Other Llama/Qwen/Mistral-style models | GGUF | Via llama.cpp bindings. | shard.yml dependencies: cogni-ml: github: skuznetsov/cogni-ml version: ~ 0.40.0 make build make spec CPU-only: make build cpu make spec cpu llama.cpp helper targets: make llama make llama env The Makefile searches common local, Homebrew, and system library locations for libllama . Override with LLAMA DIR , LLAMA BUILD , or LLAMA LIB DIR if needed. require "ml" x = ML::Autograd::Variable.rand 2, 3, requires grad: true, device: ML::Tensor::Device::CPU layer = ML::NN::Linear.new 3, 4, device: ML::Tensor::Device::CPU out = layer.forward x loss = out.mean loss.backward opt = ML::Optim::Adam.new layer.parameters opt.step opt.zero grad require "ml/llm/llama" ML::LLM.init model = ML::LLM::Model.new "path/to/model.gguf" gen = ML::LLM::Generator.new model puts gen.ask "What is Crystal?", max tokens: 100 ML::LLM.cleanup require "ml" require "ml/gguf/nomic bert" require "ml/gguf/metal backend" require "ml/metal/compute graph" ML::Metal::Device.init model = ML::GGUF::NomicBertMoE.from gguf "nomic-embed-text-v2-moe.Q5 K M.gguf", ML::GGUF::MetalBackend.new vec = model.embed "Crystal programming language" puts "dim= {vec.size}" vecs = model.embed batch "Hello", "World", "Crystal" | Kernel | Purpose | |---|---| gemm q4k.metal | Q4 K GEMV/GEMM paths for Qwen. | gemm q56k.metal | Q5 K/Q6 K/Q8 0 GEMV, top1, and helper kernels for Qwen. | gemm mm.metal | simdgroup-matrix GEMM for Q5 K/Q6 K and batched expert variants. | gemm simd.metal | Scalar SIMD GEMM fallback. | ffn qwen35.metal | Qwen FFN, add, RMSNorm, and activation helpers. | delta net.metal | Qwen 3.5 DeltaNet/recurrent kernels. | fullattn qwen35.metal | Qwen full-attention prefill/decode helpers. | attn decode qwen35.metal | Qwen gated attention decode. | attention matmul.metal | Flash-style attention matrix helpers. | bert fp16.metal | Nomic/BERT fused ops. | nn.metal | General NN ops. | | Platform | GPU | CPU | Status | |---|---|---|---| | macOS Apple Silicon | Metal | Yes | Primary target. | | macOS Intel | Metal | Yes | Supported for general Metal paths; Qwen performance focus is Apple Silicon. | | Linux | Experimental CUDA probes | Yes | Use -Dcpu only for GGUF/metadata and CUDA probe CLIs; full Qwen generation is still Metal-first. | | FreeBSD | No native Metal | Untested CPU-only | Not a primary CI target. | NVIDIA/CUDA support is currently an experimental backend-probe track, not a full decoder. The Qwen native generation path remains Metal-first. | Flag | Effect | |---|---| -Dcpu only | Disable Metal and build pure CPU paths. | -Duse gguf | Enable GGUF model loading where applicable. | MIT