{"slug": "crystal-ml-library-autograd-tensors-neural-networks-optimizers", "title": "Crystal ML Library: Autograd, Tensors, Neural Networks, Optimizers", "summary": "Cogni-ML, a Crystal machine learning library with native Apple Silicon GPU acceleration, has been released. It provides tensors, autograd, neural network layers, optimizers, and native Metal inference for GGUF models including Qwen 3.5 and nomic-embed-text-v2-moe. The library targets production-oriented AI workloads on Apple Silicon with quantized kernel support and speculative decoding.", "body_md": "Crystal machine learning library with native Apple Silicon GPU acceleration.\n\nCogni-ML is currently two things:\n\n- A general Crystal ML toolkit: tensors, autograd, NN layers, optimizers, GGUF readers, and llama.cpp bindings.\n- A native Metal inference lab for GGUF models, with production-oriented work on\n`nomic-embed-text-v2-moe`\n\nembeddings and Qwen 3.5 text generation.\n\n- Native Metal embedding pipeline for\n`nomic-embed-text-v2-moe`\n\n. - Native Qwen 3.5 9B GGUF inference path for Apple Silicon Metal.\n- Q4_K/Q5_K/Q6_K/Q8_0 quantized matmul kernels.\n- Chunked Qwen 3.5 prefill, decode wave scheduling, prompt-state cache restore, and exact speculative decode harnesses.\n- ComputeGraph wave scheduling with offset-aware barrier optimization.\n- Crystal autograd engine, NN layers, and Adam/AdamW optimizers.\n- llama.cpp FFI bindings for general GGUF model access.\n\n```\nsrc/ml/\n  core/         Tensor, Shape, MetalBuffer\n  autograd/     Variable, GradFn backward pass\n  nn/           Linear, LayerNorm, MultiHeadAttention, ViT\n  optim/        Adam/AdamW\n  llm/          llama.cpp FFI bindings\n  gguf/         GGUF reader, tokenizer, dequantization, Qwen35, NomicBertMoE\n  metal/        Device, ComputeEncoder, ComputeGraph, GraphEncoder\n```\n\nThe native Qwen path targets `Qwen3.5-9B-Q4_K_M.gguf`\n\non Apple Silicon. The code supports:\n\n- Qwen 3.5 GGUF metadata and tokenizer loading.\n- Q4_K, Q5_K, Q6_K, and Q8_0 quantized projections.\n- Full-attention layers with GQA, partial RoPE, KV cache writes, and fused output projection.\n- DeltaNet/recurrent layers with GPU-resident recurrent state and chunked prefill scan.\n- Chunked prefill with final-token top1 shortcut.\n- Decode wave scheduling to reduce command-buffer boundaries.\n- Native Qwen BPE tokenizer with an external\n`llama-tokenize`\n\nfallback for A/B. - Exact prompt-state save/restore, tokenized-prompt reuse, and longest-prefix prompt cache.\n- Exact speculative decode harnesses:\n- neural draft with Qwen 3.5 0.8B Q8_0,\n- n-gram/cache draft for repeated/generated-template text,\n- target-verifier chunks with row-batched top1 for larger accepted chunks.\n\nThe 9B Q4_K_M path is the primary verified target. Qwen 3.6 27B is a scale-up target, but it should be treated as experimental until local correctness and performance runs are completed.\n\nThe developer CLIs default to local LM Studio / llama.cpp-style paths:\n\n```\n~/.cache/lm-studio/models/lmstudio-community/Qwen3.5-9B-GGUF/Qwen3.5-9B-Q4_K_M.gguf\n~/.cache/lm-studio/models/lmstudio-community/Qwen3.5-0.8B-GGUF/Qwen3.5-0.8B-Q8_0.gguf\n~/SrcArchives/AI/llama.cpp/build/bin/llama-tokenize\n~/SrcArchives/AI/llama.cpp/build/bin/llama-bench\n```\n\nMost benchmark/probe CLIs also accept `--model`\n\n, `--target`\n\n, `--draft`\n\n, `--tokenizer-bin`\n\n, or environment overrides. `bin/qwen35_generate.cr`\n\nis intentionally a small demo and currently uses its constants at the top of the file.\n\nBuild the CPU-only GGUF/Qwen metadata smoke on Linux, CUDA hosts, or any environment where Metal is unavailable:\n\n```\ncrystal build -Dcpu_only bin/qwen35_gguf_info.cr -o build/qwen35_gguf_info\n./build/qwen35_gguf_info --model /path/to/Qwen3.5-9B-Q4_K_M.gguf\n./build/qwen35_gguf_info --model /path/to/Qwen3.5-0.8B-Q8_0.gguf --load-weights\n```\n\nThis entrypoint intentionally does not run inference. It verifies GGUF parsing, Qwen 3.5/3.6 hparams, tensor inventory, and the structured `Qwen35Weights`\n\nloader without pulling the Metal bridge into a Linux build.\n\nBuild the minimal Crystal CUDA Driver API smoke on NVIDIA/Linux hosts:\n\n```\ncrystal build bin/cuda_driver_smoke.cr -o build/cuda_driver_smoke\n./build/cuda_driver_smoke 4096\n```\n\nThe CUDA smoke is a backend boundary probe only: it links `libcuda`\n\n, loads embedded PTX, launches a vector-add kernel, and checks the result.\n\nCUDA probe code uses `src/ml/cuda/driver.cr`\n\nfor reusable CUDA context, module/function, launch, copy, synchronize, and device-buffer ownership. This is intentionally small: it owns the raw CUDA Driver API lifecycle and calls, while higher-level layer execution is still probe-local until the CUDA backend split is promoted.\nIt also provides `ML::CUDA::ResidentSequenceRunner`\n\n, a thin lifecycle facade for resident sequence probes with explicit `upload_weights`\n\n, `reset_sequence`\n\n, `run_sequence`\n\n, and `read_outputs`\n\nphases.\n`src/ml/cuda/qwen_recurrent_layer_runner.cr`\n\nis the first Qwen-specific runner extraction: it owns one recurrent layer's CUDA modules, device buffers, kernel parameters, weight upload, sequence reset, token launch graph, and output readback. `QwenRecurrentLayerRunner::Weights.load`\n\nowns GGUF tensor lookup, tensor-shape/type validation, and raw weight reads for the runner, including recurrent-layer `ffn_down`\n\ntensors stored as either Q4_K or Q6_K. CPU-reference comparison intentionally remains in the probe.\n\nBuild the first quantized CUDA correctness probe on NVIDIA/Linux hosts:\n\n```\ncrystal build -Dcpu_only bin/cuda_q8_gemv_probe.cr -o build/cuda_q8_gemv_probe\n./build/cuda_q8_gemv_probe \\\n  --model /path/to/Qwen3.5-0.8B-Q8_0.gguf \\\n  --tensor blk.0.ffn_up.weight \\\n  --kernel warp4 \\\n  --reps 100 \\\n  --warmup 10\n```\n\n`cuda_q8_gemv_probe`\n\nloads a real GGUF Q8_0 tensor, launches a Crystal-driven CUDA Driver API GEMV kernel over the raw GGUF block layout, and compares against the existing CPU `QuantMatmul`\n\nreference. `--kernel scalar`\n\nkeeps the first one-thread-per-output-row correctness kernel; the default `--kernel warp4`\n\nmaps four output rows to four warps per thread block and is the current faster probe shape. This is still a standalone backend-boundary probe, not an optimized Qwen CUDA inference path yet. The current full `qwen35_generate`\n\nCLI remains Metal-first.\n\nBuild the first Q4_K CUDA correctness probe for Qwen 9B/27B-style target tensors:\n\n```\ncrystal build -Dcpu_only bin/cuda_q4k_gemv_probe.cr -o build/cuda_q4k_gemv_probe\n./build/cuda_q4k_gemv_probe \\\n  --model /path/to/Qwen3.5-9B-Q4_K_M.gguf \\\n  --tensor blk.0.attn_gate.weight \\\n  --kernel warp4 \\\n  --reps 20 \\\n  --warmup 3\n```\n\n`cuda_q4k_gemv_probe`\n\nuses the raw GGUF Q4_K block layout (`d`\n\n, `dmin`\n\n, 12-byte packed scales/mins, 128-byte packed nibbles) and checks the CUDA output against the CPU `QuantMatmul`\n\nQ4_K reference. `--kernel scalar`\n\nkeeps the first correctness kernel; the default `--kernel warp4`\n\nmaps four output rows to four warps per block and is the current faster probe shape.\n\nBuild the Q6_K CUDA correctness/speed probe for Q4_K_M tensors that remain in Q6_K:\n\n```\ncrystal build -Dcpu_only bin/cuda_q6k_gemv_probe.cr -o build/cuda_q6k_gemv_probe\n./build/cuda_q6k_gemv_probe \\\n  --model /path/to/Qwen3.5-9B-Q4_K_M.gguf \\\n  --tensor blk.0.ffn_down.weight \\\n  --kernel warp4 \\\n  --reps 10 \\\n  --warmup 2\n```\n\n`cuda_q6k_gemv_probe`\n\ncovers the GGUF Q6_K block layout (`ql`\n\n, `qh`\n\n, signed scales, `d`\n\n) used by output/value/down projections in mixed-quant target models. Like the Q4_K/Q8_0 probes, it is a standalone backend primitive check; full CUDA Qwen execution is still a separate backend split.\n\nBuild the first GPU-resident FFN sequence probe:\n\n```\ncrystal build -Dcpu_only bin/cuda_ffn_sequence_probe.cr -o build/cuda_ffn_sequence_probe\n./build/cuda_ffn_sequence_probe \\\n  --model /path/to/Qwen3.5-9B-Q4_K_M.gguf \\\n  --layer 0 \\\n  --reps 10 \\\n  --warmup 2\n```\n\n`cuda_ffn_sequence_probe`\n\ncomposes the checked CUDA primitives as `Q4_K ffn_gate + Q4_K ffn_up -> SwiGLU -> Q6_K ffn_down`\n\nwhile keeping the input, intermediate activations, and output projection input GPU-resident. Only the final hidden vector is copied back for comparison against the CPU `QuantMatmul`\n\nFFN reference.\n\nBuild the full-attention input projection bundle probe:\n\n```\ncrystal build -Dcpu_only bin/cuda_attn_projection_probe.cr -o build/cuda_attn_projection_probe\n./build/cuda_attn_projection_probe \\\n  --model /path/to/Qwen3.5-9B-Q4_K_M.gguf \\\n  --layer 3 \\\n  --reps 10 \\\n  --warmup 2\n```\n\n`cuda_attn_projection_probe`\n\nruns `Q4_K attn_q + Q4_K attn_k + Q6_K attn_v`\n\nfrom one GPU-resident hidden vector and copies Q/K/V back only after all projections complete. It targets full-attention layers such as `blk.3`\n\nin Qwen3.5 9B.\nThe probe now routes through `ML::CUDA::QwenFullAttnProjectionRunner`\n\n, supports `--tokens N`\n\n, and keeps Q/K/V outputs GPU-resident until the final correctness readback. It is the reusable input-projection boundary for future full-attention/KV CUDA work, not a complete full-attention layer runner yet.\n\nBuild the full-attention Q/K normalization + RoPE + KV-cache boundary probe:\n\n```\ncrystal build -Dcpu_only bin/cuda_full_attn_kv_probe.cr -o build/cuda_full_attn_kv_probe\n./build/cuda_full_attn_kv_probe \\\n  --model /path/to/Qwen3.5-9B-Q4_K_M.gguf \\\n  --layer 3 \\\n  --tokens 4 \\\n  --start-pos 2 \\\n  --max-seq 12\n```\n\n`cuda_full_attn_kv_probe`\n\nnow routes through `ML::CUDA::QwenFullAttnLayerRunner`\n\n, a residual-hidden-to-final-hidden wrapper around the projection runner and `ML::CUDA::QwenFullAttnKVRunner`\n\n. The projection runner can apply the initial `attn_norm`\n\non CUDA from residual hidden states before Q/K/V projection; Q is split into normalized/RoPE'd Q and gate, K is RMSNormed/RoPE'd, K/V rows are appended to a CUDA-resident cache at `start_pos`\n\n, a correctness-first serial CUDA kernel computes GQA scores, softmax, value reduction, and Q-gate multiplication, the resident gated attention output is projected through `attn_output.weight`\n\n, and the layer tail runs residual add, post-attention RMSNorm, FFN gate/up/SwiGLU/down, and final residual. It checks Q, gate, K, gated attention output, projected attention output, final hidden, K-cache, and V-cache against the CPU Qwen reference. This is now a clean one-layer semantics probe with device input/output hooks for mixed-stack composition, but it is not yet an end-to-end Linux decode path: full/recurrent stack scheduling, logits/top1, tokenizer/sampling, restored nonzero prefix KV, and a faster attention kernel remain separate gates.\n\nBuild the Q5_K CUDA recurrent-QKV probe:\n\n```\ncrystal build -Dcpu_only bin/cuda_q5k_gemv_probe.cr -o build/cuda_q5k_gemv_probe\n./build/cuda_q5k_gemv_probe \\\n  --model /path/to/Qwen3.5-9B-Q4_K_M.gguf \\\n  --tensor blk.0.attn_qkv.weight \\\n  --reps 10 \\\n  --warmup 2\n```\n\n`cuda_q5k_gemv_probe`\n\ncovers the GGUF Q5_K block layout used by recurrent-layer combined `attn_qkv.weight`\n\ntensors in the current Qwen3.5 9B Q4_K_M file.\n\nBuild the recurrent-layer projection bundle probe:\n\n```\ncrystal build -Dcpu_only bin/cuda_recurrent_projection_probe.cr -o build/cuda_recurrent_projection_probe\n./build/cuda_recurrent_projection_probe \\\n  --model /path/to/Qwen3.5-9B-Q4_K_M.gguf \\\n  --layer 0 \\\n  --reps 10 \\\n  --warmup 2\n```\n\n`cuda_recurrent_projection_probe`\n\nruns `Q5_K attn_qkv + Q4_K attn_gate + Q4_K ssm_alpha + Q4_K ssm_beta`\n\nfrom one GPU-resident hidden vector and copies the four outputs back only after all kernels complete. It is the first CUDA recurrent projection-bundle proof; DeltaNet recurrence, convolution, state updates, and `ssm_out`\n\nremain separate work.\n\nBuild the synthetic DeltaNet output slice probe:\n\n```\ncrystal build -Dcpu_only bin/cuda_deltanet_output_probe.cr -o build/cuda_deltanet_output_probe\n./build/cuda_deltanet_output_probe \\\n  --model /path/to/Qwen3.5-9B-Q4_K_M.gguf \\\n  --layer 0 \\\n  --reps 10 \\\n  --warmup 2\n```\n\n`cuda_deltanet_output_probe`\n\nruns a synthetic CUDA DeltaNet state update, applies post RMSNorm/SiLU gating on GPU, and feeds the result directly into the real Q4_K `ssm_out.weight`\n\nprojection. It is a stateful boundary probe, not a full recurrent layer: recurrent conv prep, alpha/beta transforms, residuals, and FFN remain separate work.\n\nBuild the recurrent prep/output slice probe:\n\n```\ncrystal build -Dcpu_only bin/cuda_recurrent_prep_output_probe.cr -o build/cuda_recurrent_prep_output_probe\n./build/cuda_recurrent_prep_output_probe \\\n  --model /path/to/Qwen3.5-9B-Q4_K_M.gguf \\\n  --layer 0 \\\n  --tokens 4 \\\n  --reps 10 \\\n  --warmup 2\n```\n\n`cuda_recurrent_prep_output_probe`\n\nnow composes one full recurrent layer token slice: input RMSNorm, real recurrent projection bundle (`attn_qkv`\n\n, `attn_gate`\n\n, `ssm_alpha`\n\n, `ssm_beta`\n\n), recurrent conv prep, alpha/beta transforms, DeltaNet, post RMSNorm/SiLU, Q4_K `ssm_out`\n\n, residual add, post-attention RMSNorm, Q4_K FFN gate/up, SwiGLU, Q6_K FFN down, and final residual. `--tokens N`\n\nruns a GPU-resident sequence through persistent conv/SSM state and compares all token outputs plus final recurrent states against the CPU reference. The probe now separates one-time weight upload from per-sequence input/state reset and prints `weight_upload_ms`\n\n; timed `cuda_ms_per_token`\n\nexcludes the persistent weight upload. GGUF recurrent-layer tensor loading is routed through `QwenRecurrentLayerRunner::Weights.load`\n\n, so the probe no longer manually passes every raw tensor into the runner constructor. It is still a standalone one-layer probe, not an end-to-end Linux decoder.\n\nBuild the recurrent multi-layer stack scaffold:\n\n```\ncrystal build -Dcpu_only bin/cuda_recurrent_stack_probe.cr -o build/cuda_recurrent_stack_probe\n./build/cuda_recurrent_stack_probe \\\n  --model /path/to/Qwen3.5-9B-Q4_K_M.gguf \\\n  --layers 0,2,4 \\\n  --tokens 2\n```\n\n`cuda_recurrent_stack_probe`\n\nchains multiple `QwenRecurrentLayerRunner`\n\ninstances and compares the final hidden sequence plus each layer's recurrent conv/SSM state against the CPU reference. The default path hands each recurrent layer's CUDA output buffer directly to the next layer's CUDA input; `--host-handoff`\n\nkeeps the older host-copy route as a debug oracle. This is still a recurrent-only scaffold, not an end-to-end Linux decoder.\n\nBuild the mixed recurrent/full-attention CUDA stack probe:\n\n```\ncrystal build -Dcpu_only bin/cuda_mixed_stack_probe.cr -o build/cuda_mixed_stack_probe\n./build/cuda_mixed_stack_probe \\\n  --model /path/to/Qwen3.5-9B-Q4_K_M.gguf \\\n  --layers 0,1,2,3,4 \\\n  --tokens 2 \\\n  --start-pos 2 \\\n  --max-seq 12\n```\n\n`cuda_mixed_stack_probe`\n\ncomposes `QwenRecurrentLayerRunner`\n\nand `QwenFullAttnLayerRunner`\n\nin model layer order with device-resident hidden handoff across recurrent/full-attention boundaries, then runs `QwenOutputHeadRunner`\n\nfor output RMSNorm, quantized lm-head projection, and resident top1. The layer/head loop is now owned by `ML::CUDA::QwenMixedStackRunner`\n\n, which is the first model-slice decode-state object for CUDA. By default the probe copies back only the CUDA top1 id/value plus hidden/state debug outputs; pass `--read-logits`\n\nto also copy full logits for attribution, and `--profile-phases`\n\nto insert per-layer/head synchronizations and print attribution lines. It compares the final hidden sequence, top1, recurrent conv/SSM states, and full-attention KV cache rows against the CPU reference. This is the first mixed-stack CUDA correctness scaffold through resident top1; it still stops before tokenizer/sampling, repeated full-model decode ownership, and an optimized topK/sampling kernel. The current resident top1 is a simple two-phase partial-scan/reduce kernel: correct, but not yet promoted as a speed-optimized head.\n\nExperimental CUDA exact cache-replay fast lane:\n\n```\ncrystal build --release --no-debug -Dcpu_only \\\n  bin/cuda_mixed_stack_probe.cr \\\n  -o build/cuda_mixed_stack_probe\n\n./build/cuda_mixed_stack_probe \\\n  --model /path/to/Qwen3.5-9B-Q4_K_M.gguf \\\n  --all-layers \\\n  --max-seq 256 \\\n  --greedy-loop-tokens 64 \\\n  --greedy-loop-probe-chunk-gamma 16 \\\n  --greedy-loop-probe-chunk-active-verify \\\n  --greedy-loop-probe-ngram \\\n  --greedy-loop-probe-ngram-source-history \"$SOURCE_TOKEN_IDS\" \\\n  --greedy-loop-probe-ngram-replay-start 1 \\\n  --greedy-loop-probe-ngram-cursor-only \\\n  --greedy-loop-probe-ngram-trusted-source \\\n  --greedy-loop-probe-ngram-schedule 64 \\\n  --skip-debug-readback\n```\n\nThis path is exact verification of a validated source/cache cursor, not a\ngeneral-purpose sampler. It first checks that the live prefix matches the source\nhistory at the replay cursor. If the prefix gate fails, the active verifier path\nis disabled before any proposals are trusted and the probe falls back to plain\ngreedy target decode. If the gate passes, proposal chunks are verified through\nthe same resident CUDA stack and rejected chunks restore the exact target state.\nUse smaller schedules such as `4,4,8,16`\n\nfor weaker proposal sources where early\nreject economics matter; use bulk schedules such as `64`\n\nonly for artifact-level\ntrusted replay where the source history was produced by the same model/cache\ncontract and every proposed token is still target-verified before commit.\n\nTrusted artifact restore lower-bound:\n\n```\n./build/cuda_mixed_stack_probe \\\n  --model /path/to/Qwen3.5-9B-Q4_K_M.gguf \\\n  --all-layers \\\n  --max-seq 256 \\\n  --known-replay-history \"$SOURCE_TOKEN_IDS\" \\\n  --known-replay-start 1 \\\n  --known-replay-tokens 64 \\\n  --known-replay-trusted-artifact-restore \\\n  --skip-debug-readback\n```\n\nFor a conservative host-backed artifact simulation, add:\n\n```\n  --known-replay-trusted-artifact-host-restore\n```\n\nTo snapshot only KV rows up to the restored cursor instead of the full\n`max_seq`\n\ncache, use:\n\n```\n  --known-replay-trusted-artifact-live-kv\n```\n\nTo include diagnostic artifact file IO and hash timing, add:\n\n```\n  --known-replay-trusted-artifact-io-probe \\\n  --known-replay-trusted-artifact-io-path /path/to/artifact.bin\n```\n\nThis is a different contract from verified replay. It simulates a cache artifact that already contains the exact token span and the post-span decode state for the same model/config/source hash. The timed region restores that state and emits cached tokens; it does not recompute the verifier body. Use it only for session/cache artifacts whose model hash, tokenizer hash, prefix hash, token span, and state artifact hash have been validated.\n\nThe default trusted-artifact probe snapshots/restores state inside device memory\nand is a pure lower bound. The host-backed variant copies the complete decode\nstate to host memory, poisons the runner, then times host-to-device restore. On\nthe RTX 5060 Ti Qwen3.5-9B snapshot with `max_seq=128`\n\n, that state is\n`61,079,552`\n\nbytes; restore costs `12.259ms`\n\nfor a 64-token artifact and\n`12.331ms`\n\nfor a non-zero start9/24-token artifact. This is still not full\nproduction IO: durable lookup, disk reads, hashing, and live-length KV trimming\nremain outside the measured region.\n\nThe live-KV variant preserves exact output while reducing only the full-attention\ncache portion. On the same host/model, start1/gen64 stores `56,885,248`\n\nbytes\nand restores in `11.358ms`\n\n; start9/gen24 stores `54,788,096`\n\nbytes and restores\nin `10.984ms`\n\n. The modest delta is a useful finding: recurrent DeltaNet state,\nnot KV capacity, dominates this short-context artifact.\n\nOn the persistent reefy storage path, the live-KV start1/gen64 artifact breaks\ndown into `52,690,944`\n\nrecurrent-state bytes and `4,194,304`\n\nKV bytes. The\ndiagnostic file path measured write `17.695ms`\n\n, read `42.046ms`\n\n, SHA-256\nverification `59.556ms`\n\n, and H2D restore `13.003ms`\n\n. For start9/gen24, the\nartifact is `52,690,944`\n\nrecurrent bytes plus `2,097,152`\n\nKV bytes; write/read/\nhash/restore measured `18.249/42.994/53.814/12.825ms`\n\n. The SHA-256 timing uses\nthe host `sha256sum`\n\n/`shasum`\n\ntool to avoid adding OpenSSL linkage to the CUDA\nprobe, so treat it as a product-boundary diagnostic rather than a kernel metric.\nAn experimental contiguous-read reconstruction reduced start1/gen64 read time\nfrom `42.046ms`\n\nto `36.310ms`\n\nwhile preserving `64/64`\n\nexact output, but start9/\ngen24 remained noisy (`43.618ms`\n\nread). This points to some avoidable allocation/\nfragmentation cost, but not enough to change the main conclusion: cache artifacts\nneed async prefetch or a resident artifact service to stay off the decode\ncritical path.\nA first scalar CPU recurrent-state block-INT8 codec diagnostic compresses the\n`52,690,944`\n\nrecurrent bytes to `13.18-14.00MB`\n\n(`25.0-26.6%`\n\n) depending on\nblock size. On start1/gen64, block sizes `64/256/1024/4096`\n\nmeasured relative\nRMSE `0.008798/0.012300/0.015476/0.019087`\n\n; encode/decode stayed around\n`457-469ms`\n\n/ `121-125ms`\n\n. A follow-up restore gate now decodes the block-INT8\nrecurrent buffers back into a host snapshot, restores that state, and runs one\ncontinuation token against an exact uncompressed restored state. On the RTX 5060\nTi host, block sizes `64/256/1024/4096`\n\npreserved the same next top1 on\nstart9/gen24, and block `256/4096`\n\nalso preserved the same next top1 on a later\nstart33/gen8 cursor. The multi-step source-aligned gate is stronger: block4096\npreserved `16/16`\n\ncontinuation top1 ids on start9/gen24, and block256/block4096\npreserved `8/8`\n\ncontinuation top1 ids on start33/gen8. The free-run greedy gate\nnow starts both exact-restored and decoded-INT8-restored states from the same\ntoken and feeds back each path's own generated top1; block256 preserved `8/8`\n\nfree-run ids on start33/gen8, and block4096 preserved `16/16`\n\nfree-run ids on\nstart9/gen24. A wider six-cursor free-run sweep changed the conclusion: block256\npreserved `16/16`\n\nparity on all six tested cursors, and block1024 also preserved\n`16/16`\n\non those cursors, but block4096 drifted on later cursors (`15/16`\n\n,\n`15/16`\n\n, and `6/16`\n\n). So block4096 is not a trusted restore format; block1024 is\nthe current best compression/parity trade-off candidate. This is still not a\nproduction codec: the current scalar CPU codec is too slow for the critical\npath, and these gates cover one prompt/history, not the full prompt distribution.\n\nBuild the Metal bridge once:\n\n```\nmake build/bridge.o\n```\n\nBuild the practical generation demo:\n\n```\ncrystal build --release --no-debug \\\n  --link-flags=\"$(pwd)/build/bridge.o -framework Metal -framework Foundation -lc++\" \\\n  bin/qwen35_generate.cr \\\n  -o build/qwen35_generate\n```\n\nRun greedy generation:\n\n```\n./build/qwen35_generate \"The capital of France is\" 64\n```\n\nEnable exact n-gram speculative decode for repeated text:\n\n```\nQWEN35_NGRAM_DECODE=1 ./build/qwen35_generate \"The capital of France is\" 64\n```\n\nUse the conservative automatic decode policy:\n\n```\nQWEN35_DECODE_POLICY=auto ./build/qwen35_generate \"The capital of France is\" 64\n```\n\n`auto`\n\nis the product-safe proposal-aware profile: it uses exact n-gram/cache\nproposals only when they are large enough to amortize verification, enables the\ncandidate-shape risk gate plus a runtime corridor detector for untrusted suffix\nreplay, and otherwise falls back to exact target decoding without invoking the\nneural draft model.\n\nEnable exact neural speculative decode with the Qwen 3.5 0.8B draft:\n\n```\nQWEN35_SPECULATIVE_DECODE=1 \\\nQWEN35_HEAD_FULL_ROWS_GUARDED=1 \\\n./build/qwen35_generate \"The capital of France is\" 64\n```\n\nExperimental same-weight proposal-route memory:\n\n```\ncrystal build --release --no-debug \\\n  bin/qwen35_proposal_route_memory.cr \\\n  -o build/qwen35_proposal_route_memory\n\nQWEN35_ROUTE_CAL_ROOT=/tmp/qwen35_route_memory \\\nQWEN35_ROUTE_CAL_GEN=16 \\\nQWEN35_ROUTE_CAL_GAMMA=4 \\\nQWEN35_ROUTE_CAL_UPDOWN_RANK=4 \\\nQWEN35_ROUTE_CAL_UPDOWN_LAYERS=0,2,4 \\\nscripts/qwen35_proposal_route_calibrate.sh\n```\n\nThe route cache stores certified proposal-body choices such as `baseline`\n\nor\n`pca_updown`\n\nfor the GPU self-spec probe path. It is intentionally not a\n`qwen35_generate`\n\ndecode-policy switch yet: the product generator currently has\ngreedy, n-gram/cache, external-draft speculative, and GGUF-MTP paths, while the\nPCA-updown same-weight proposal body lives in\n`bin/qwen35_deltanet_fixed_basis_probe.cr`\n\n. Use route memory to avoid repeating\nonline route calibration in that probe corridor; do not count it as a product\ngeneration speedup until the same verifier/proposal corridor is wired into the\nnormal generator. `qwen35_generate`\n\ncan resolve the route table with\n`QWEN35_SELF_SPEC_ROUTE_MEMORY_ROOT`\n\nfor diagnostics and future wiring, but it\nprints `product_self_spec=unsupported`\n\nand leaves the decode path unchanged.\nIf `QWEN35_SELF_SPEC_UPDOWN_ADAPTERS_PATH`\n\nis also set, the generator validates\nthe referenced PCA-updown adapter artifact against the selected route layers and\nrank through `ML::GGUF::Qwen35SelfSpecPlan`\n\n, then reports\n`plan=pca_updown|invalid_adapter|baseline|route_miss`\n\nand\n`adapter_artifact=valid|invalid`\n\nwithout executing the adapter.\nThe PCA-updown FFN adapter data path is now shared in\n`src/ml/gguf/qwen35_ffn_updown_adapter.cr`\n\n: it owns the centered low-rank\nprojection math, Hadamard/symmetric quant-dequant helpers, and the\n`qwen35_ffn_updown_adapter_v1`\n\nartifact dump/load format. The heavy\nself-spec scheduler is still probe-local.\n\nManual route lookup/seed example:\n\n```\n./build/qwen35_proposal_route_memory \\\n  --root /tmp/qwen35_route_memory \\\n  --prompt \"def square(x): return x * x\\n\" \\\n  --route-key code_square \\\n  --route pca_updown \\\n  --rank 4 \\\n  --layers 0,2,4\n```\n\nEnable Qwen chat-template prompting and XML-style function calls:\n\n```\nQWEN35_CHAT=1 \\\nQWEN35_CHAT_SYSTEM=\"You are a tool-using assistant.\" \\\nQWEN35_TOOLS_JSON='[{\"type\":\"function\",\"function\":{\"name\":\"get_weather\",\"description\":\"Get weather for a city\",\"parameters\":{\"type\":\"object\",\"properties\":{\"city\":{\"type\":\"string\"}},\"required\":[\"city\"]}}}]' \\\n./build/qwen35_generate \"Weather in Paris?\" 64\n```\n\n`QWEN35_TOOLS_JSON`\n\nuses the tool schema shape embedded in the Qwen\n`tokenizer.chat_template`\n\n. Qwen 3.5/3.6 GGUFs use an XML-ish tool-call format,\nfor example `<tool_call><function=...><parameter=...>...`\n\n, not a constrained\nJSON-schema decoder. The CLI renders the model prompt with the Qwen chat tokens,\nuses the rendered prompt for tokenization and prompt-cache keys, and prints a\nparsed JSON summary when generated text contains Qwen `<tool_call>`\n\nblocks.\n\nFor harnesses that expect normal JSON function-calling, keep the model-facing Qwen XML template and normalize only at the host boundary:\n\n```\nQWEN35_CHAT=1 \\\nQWEN35_TOOL_RESPONSE_JSON=simple \\\nQWEN35_TOOLS_JSON='[{\"type\":\"function\",\"function\":{\"name\":\"read_file\",\"description\":\"Read a file\",\"parameters\":{\"type\":\"object\",\"properties\":{\"path\":{\"type\":\"string\"}},\"required\":[\"path\"]}}}]' \\\n./build/qwen35_generate \"Read src/foo.cr\" 64\n```\n\nThe emitted `=== Tool response JSON ===`\n\nblock uses the CrystalBall-compatible\nshape. When `QWEN35_TOOLS_JSON`\n\nis available, argument normalization is\nschema-aware for basic scalar types, so string fields stay strings while\ninteger/number/boolean fields are emitted as typed JSON values:\n\n```\n{\"content\":null,\"tool_calls\":[{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/foo.cr\"}}]}\n```\n\nSet `QWEN35_TOOL_RESPONSE_JSON=openai`\n\nto emit OpenAI-style wrappers with\n`id`\n\n, `type`\n\n, and `function.arguments`\n\nas a JSON string. Multi-turn harnesses\ncan pass OpenAI-style messages through `QWEN35_MESSAGES_JSON`\n\n; assistant\n`tool_calls`\n\nare rendered back into Qwen XML blocks and `tool`\n\nmessages are\nrendered as Qwen `tool`\n\nrole messages.\n\n`../crystal_ball`\n\ncan use this backend through its `CogniQwen`\n\nprovider:\n\n```\ncrystal build --release bin/qwen35_generate -o build/qwen35_generate\ncd ../crystal_ball\nCB_COGNI_QWEN=1 \\\nCB_COGNI_QWEN_BIN=../cogni-ml/build/qwen35_generate \\\ncrystal run src/cli.cr\n```\n\nThe lightweight adapter is useful for testing prompt/render and output parsing without loading the model:\n\n```\nprintf '<tool_call>\\n<function=read_file>\\n<parameter=path>\\nsrc/foo.cr\\n</parameter>\\n</function>\\n</tool_call>\\n' \\\n  | crystal run bin/qwen35_tool_json_adapter.cr -- --parse-output --format=simple\n\nprintf '{\"messages\":[{\"role\":\"user\",\"content\":\"Read src/foo.cr\"}],\"tools\":[]}' \\\n  | crystal run bin/qwen35_tool_json_adapter.cr -- --render-request\n```\n\nEnable exact prompt cache:\n\n```\nQWEN35_PROMPT_CACHE=1 \\\nQWEN35_SESSION_ID=demo \\\n./build/qwen35_generate \"The capital of France is\" 64\n```\n\nWith prompt cache enabled, `qwen35_generate`\n\nnow caches two independent\nartifacts:\n\n- tokenized prompt text, so repeated prompts avoid the older external tokenizer process path;\n- exact prompt state, including optional full-prompt next-token metadata for long generations.\n\nThe prompt-state lookup is longest-prefix based. If the new prompt extends a\ncached prompt, the CLI restores the verified prefix state and exact-replays only\nthe uncached suffix before decode. A local product smoke on Qwen3.5 9B Q4_K_M\nrestored `12/14`\n\nprompt tokens, replayed `2`\n\n, skipped normal prompt prefill\n(`prefill_ms=0.0`\n\n), and used `cache_route=prompt_state_restore`\n\n.\n`QWEN35_PROMPT_CACHE_FULL_HIT_MIN_GEN`\n\ncontrols when a full-prompt hit can use\nstored next-token metadata directly; the default favors longer generations so\nshort requests do not merely shift first model work from `prefill_ms`\n\nto\n`decode_ms`\n\n.\n\nWhen `QWEN35_PROMPT_CACHE_FAST_FORWARD=1`\n\nand source-history cache is enabled,\nthe CLI also writes a direct per-key output fast-forward certificate for fully\ngenerated spans. On a repeated same-session terminal request, this lets\n`qwen35_generate`\n\nvalidate model/session/prompt/output hashes and emit cached\ntoken ids/text before opening the GGUF. The older tokenized-prompt +\nsource-history + manifest scan remains as a fail-closed fallback for legacy or\ntampered direct certificates.\n\nFast-forward state artifacts can also be stored in the guarded compressed v2\nformat. Set `QWEN35_PROMPT_CACHE_ARTIFACT_CODEC=recurrent-bf16`\n\nto compress\nrecurrent DeltaNet state while keeping KV rows raw; Metal restores these\nvalidated artifacts through the mmap encoded path. `recurrent-int8`\n\nremains\nexplicitly gated by `QWEN35_PROMPT_CACHE_METAL_INT8_RESTORE=1`\n\n.\n\nThe native tokenizer is the default. Set `QWEN35_NATIVE_TOKENIZER_OFF=1`\n\nonly\nwhen comparing against the external `llama-tokenize`\n\nbootstrap path.\n\nEach generation ends with a compact request-phase summary:\n\n```\nrequest summary: total_ms=... model_load_ms=... draft_load_ms=... tokenize_ms=... token_cache_hit=... state_prepare_ms=... source_history_lookup_ms=... cache_restore_ms=... prefill_ms=... decode_ms=... source_history_save_ms=... prompt_tokens=... output_tokens=...\n```\n\nUse `total_ms`\n\nfor one-shot CLI latency. Use `decode_ms / output_tokens`\n\nonly\nwhen comparing decoder loops after the model, tokenizer, prompt cache, and\nstate setup costs are already accounted for.\n\nMeasure warm resident-process request latency without changing product CLI semantics:\n\n```\ncrystal build --release --no-debug \\\n  --link-flags=\"$(pwd)/build/bridge.o -framework Metal -framework Foundation -framework MetalPerformanceShaders -lc++\" \\\n  bin/qwen35_warm_request_probe.cr \\\n  -o build/qwen35_warm_request_probe\n\n./build/qwen35_warm_request_probe \\\n  --gen 16 \\\n  --requests 5 \\\n  --warmups 1 \\\n  --quiet \\\n  \"The capital of France is\"\n```\n\nThis probe loads model/tokenizer once, runs explicit warmup requests outside the\nmeasured set, then reports per-request total/tokenize/state-prepare/prefill/decode\ntimings. Use it to evaluate daemon/server-mode economics; do not mix it with\none-shot `qwen35_generate`\n\ntotals.\n\nTo isolate exact source-history replay economics without one-shot CLI cache, disk, or lazy Metal compile tax, run the resident replay mode:\n\n```\n./build/qwen35_warm_request_probe \\\n  --source-replay \\\n  --gen 64 \\\n  --requests 3 \\\n  --warmups 1 \\\n  --quiet \\\n  \"alpha beta gamma delta alpha beta gamma delta\"\n\n./build/qwen35_warm_request_probe \\\n  --source-replay \\\n  --metal-profile \\\n  --gen 64 \\\n  --requests 1 \\\n  --warmups 1 \\\n  --quiet \\\n  \"alpha beta gamma delta alpha beta gamma delta\"\n```\n\n`--source-replay`\n\nseeds one exact generated span, restores the prompt state\ninside the same resident process, and verifies the known span in one chunk with\nthe same tail-skip contract as source-history cache replay. It is a\ncache/session replay measurement, not plain generation throughput.\n\nTo measure the real prompt-cache `Store`\n\nrestore path in the same resident\nprocess, use `--prompt-cache-replay`\n\n. `--resident-states=1`\n\nenables the hot\nin-memory restored-state template path:\n\n```\n./build/qwen35_warm_request_probe \\\n  --prompt-cache-replay \\\n  --resident-states 1 \\\n  --gen 64 \\\n  --requests 3 \\\n  --warmups 1 \\\n  --quiet \\\n  \"alpha beta gamma delta alpha beta gamma delta\"\n```\n\nTo measure validated session-cache fast-forward, use\n`--prompt-cache-fast-forward`\n\n. This mode seeds an exact span once, stores the\nstate after the processed part of that span, validates the full token-history\nhash, restores the cached state, and emits the cached output ids without running\nthe verifier body:\n\n```\n./build/qwen35_warm_request_probe \\\n  --prompt-cache-fast-forward \\\n  --resident-states 1 \\\n  --reuse-request-state \\\n  --gen 64 \\\n  --requests 3 \\\n  --warmups 1 \\\n  --quiet \\\n  \"alpha beta gamma delta alpha beta gamma delta\"\n```\n\nThis is a cache-hit/session fast-forward measurement, not generation\nthroughput. It is exact only when the cached span and state artifact are\nvalidated for the same model, tokenizer, prompt/session tokens, and runtime\ncontract. `--reuse-request-state`\n\nmodels a daemon/server state pool: cached\nstate restore overwrites the same prepared destination buffers each request\ninstead of allocating a fresh destination state. On a local M2 Max Qwen3.5 9B\nQ4_K_M smoke, resident fast-forward measured `~4.5 ms`\n\ntotal for 64 cached\ntokens (`~0.07 ms/tok`\n\n) before request-state pooling; a later 16-token\nBF16/live-KV smoke measured p50 total `~0.6-1.0 ms`\n\nwith request-state reuse.\nThe same Store source replay path remains slower because it still runs the exact\nbulk verifier body.\n\nTo measure the terminal direct-output certificate path inside a resident\nprocess, use `--prompt-cache-direct-output`\n\n. This mode seeds the same exact span\nand certificate once, then times only `Store#lookup_output_fast_forward`\n\nvalidation and cached id emission; it intentionally performs no state restore,\nprefill, or decoder work:\n\n```\n./build/qwen35_warm_request_probe \\\n  --prompt-cache-direct-output \\\n  --gen 16 \\\n  --requests 7 \\\n  --warmups 2 \\\n  --quiet \\\n  \"The capital of France is\"\n```\n\nOn a local M2 Max Qwen3.5 9B Q4_K_M smoke, this measured p50 `0.009 ms`\n\ntotal for a 16-token cached span after warmup. The same build measured hot\nBF16/live-KV state fast-forward with `--resident-states=1`\n\nand\n`--reuse-request-state`\n\nat p50 `0.587 ms`\n\non the same prompt/settings. Treat\ndirect-output certificate hits as the terminal repeated-output path; use state\nfast-forward only when the caller needs a continuation state after the cached\nspan.\n\nTo exercise the resident serving route order directly, use\n`--prompt-cache-serving-route`\n\n. Terminal requests try the direct output\ncertificate first. If the caller requires continuation state after the cached\nspan, add `--serving-route-continuation`\n\n; this bypasses terminal id emission and\nuses the validated state fast-forward corridor instead:\n\n```\n./build/qwen35_warm_request_probe \\\n  --prompt-cache-serving-route \\\n  --gen 16 \\\n  --requests 7 \\\n  --warmups 2 \\\n  --quiet \\\n  \"The capital of France is\"\n\n./build/qwen35_warm_request_probe \\\n  --prompt-cache-serving-route \\\n  --serving-route-continuation \\\n  --resident-states 1 \\\n  --reuse-request-state \\\n  --artifact-codec recurrent-bf16 \\\n  --live-kv-artifacts \\\n  --gen 16 \\\n  --requests 7 \\\n  --warmups 2 \\\n  --quiet \\\n  \"The capital of France is\"\n```\n\nThe request summaries include `route=direct_output`\n\nor\n`route=state_fast_forward_continuation`\n\n. A local M2 Max smoke measured p50\n`0.010 ms`\n\nfor the terminal direct route and p50 `0.661 ms`\n\nfor the\ncontinuation-state route on the same 16-token cached span.\n`--serving-route-active-cursor`\n\nis an upper-bound server-shape probe: it\nprewarms the continuation state once and then measures the already-owned\nactive-session handoff, avoiding the reusable-cache copy that a shared Store\nmust perform for isolation.\nFor matrix gates, set `QWEN35_MATRIX_MODE=serving-route`\n\n; the TSV includes a\n`routes`\n\ncolumn so policy changes are visible in apples-to-apples comparisons.\nFor fail-closed diagnostics, `--serving-route-direct-miss`\n\nomits the direct\noutput certificate while keeping the exact-known-span artifact. Terminal\nrequests then fall back to\n`route=source_history_direct_output_fallback`\n\nwithout restoring state; requests\nthat need continuation state still use the validated state corridor.\nThe route decision is also available to product code as\n`ML::GGUF::Qwen35ServingRoute.serve_exact_cached_span`\n\nin\n`src/ml/gguf/qwen35_serving_route.cr`\n\n.\nFor resident servers, `ML::GGUF::Qwen35ResidentSession`\n\nin\n`src/ml/gguf/qwen35_resident_session.cr`\n\nowns route counters and an optional\nactive continuation cursor while keeping shared Store restores copy-safe.\n\nFor a repeatable raw-vs-compressed cache-artifact gate, use the matrix runner:\n\n```\nscripts/qwen35_cache_artifact_matrix.sh\n```\n\nBy default it builds/uses `/tmp/qwen35_warm_request_probe_matrix`\n\n, runs five\nprompt classes through `--prompt-cache-fast-forward`\n\n, and reports tab-separated\nrows with `mode`\n\n, artifact codec, live-KV policy, request totals, and phase\ntimings. Useful overrides:\n\n```\nQWEN35_MATRIX_PROMPT_LIMIT=1 \\\nQWEN35_MATRIX_REQUESTS=1 \\\nQWEN35_MATRIX_WARMUPS=0 \\\nQWEN35_MATRIX_CODECS=\"raw recurrent-bf16\" \\\nQWEN35_MATRIX_LIVE_KV=\"0 1\" \\\nQWEN35_MATRIX_REUSE_REQUEST_STATE=1 \\\nscripts/qwen35_cache_artifact_matrix.sh\n\nQWEN35_MATRIX_MODE=direct-output \\\nQWEN35_MATRIX_REQUESTS=7 \\\nQWEN35_MATRIX_WARMUPS=2 \\\nscripts/qwen35_cache_artifact_matrix.sh\n```\n\n`QWEN35_MATRIX_MODE=direct-output`\n\ncollapses the artifact axes to\n`codec=direct`\n\nand `live_kv=na`\n\n, because this mode measures only the resident\ndirect output-certificate validator and cached id emission.\n\nUse the matrix for cache-artifact economics only. It does not measure first-run prefill or plain generation throughput.\n\nFor prefill/decode profiling logs, use the graph-atlas helper to turn\n`Qwen35Metal.Profile`\n\nreports into ranked LTP/WBA windows:\n\n```\nQWEN35_PREFILL_PHASE_PROFILE=1 /tmp/qwen35_prefill_attribution \\\n  --prompt=64 --warmup=1 --reps=1 --prepare-state \\\n  --load-warning-threshold=0 --load-total-warning-threshold=0 \\\n  > /tmp/qwen35_prefill_profile.log\n\nscripts/qwen35_profile_atlas.cr /tmp/qwen35_prefill_profile.log --top=8\n```\n\nThe atlas is an attribution aid, not a benchmark by itself. Phase profiling intentionally changes command-buffer boundaries, so use paired wall timing before promoting any kernel or scheduling change.\n\nUseful Qwen environment switches:\n\n| Variable | Effect |\n|---|---|\n`QWEN35_PROMPT_CACHE=1` |\nEnable exact prompt-state cache lookup/save in `qwen35_generate` . |\n`QWEN35_PROMPT_CACHE_ROOT=/path` |\nOverride prompt-cache artifact root. |\n`QWEN35_PROMPT_TOKEN_CACHE_OFF=1` |\nDisable tokenized-prompt cache lookup/save while keeping prompt-state cache enabled. |\n`QWEN35_PROMPT_CACHE_FAST_FORWARD=1` |\nWith prompt cache and source-history enabled, save and use validated post-span state artifacts so exact session-cache hits can emit cached spans without verifier recompute. Default off; falls back when source prefix, token hash, or artifact validation fails. |\n`QWEN35_PROMPT_CACHE_PREWEIGHT_FAST_FORWARD_OFF=1` |\nDiagnostic switch: disable pre-weight terminal fast-forward exits so `qwen35_generate` can exercise the post-weight serving-route/state-restore path. Default off; normal product routing still tries zero-GGUF/zero-weight terminal hits first. |\n`QWEN35_PROMPT_CACHE_ARTIFACT_CODEC=recurrent-bf16` |\nStore validated fast-forward state artifacts in compressed v2 BF16 recurrent format. KV rows remain raw. Default is raw; `recurrent-int8` is available only with the explicit Metal INT8 gate. |\n`QWEN35_PROMPT_CACHE_ARTIFACT_CODEC_BLOCK=8` |\nBlock size for `recurrent-int8` prompt-cache artifacts. Ignored for raw/BF16. |\n`QWEN35_PROMPT_CACHE_LIVE_KV_ARTIFACTS=1` |\nStore prompt-cache artifacts in v3 live-KV format: full recurrent state plus only live full-attention KV rows. This is opt-in while broader validation continues. |\n`QWEN35_PROMPT_CACHE_METAL_INT8_RESTORE=1` |\nExplicitly allow Metal restore of validated `recurrent-int8` artifacts. Default off because INT8 remains approximate and needs stronger prompt/session validation than BF16. |\n`QWEN35_PROMPT_CACHE_FULL_HIT_MIN_GEN=64` |\nMinimum requested generation length before a full-prompt cache hit can skip suffix replay and use stored next-token metadata. Lower values are useful for experiments but can move first model work into `decode_ms` without improving total wall time. |\n`QWEN35_PROMPT_CACHE_RESIDENT_STATES=0` |\nNumber of restored prompt-cache states to keep hot in a resident process. `0` disables the in-memory state cache. Positive values avoid rereading and redecompressing `.qkv` artifacts on repeated same-process session hits. |\n`QWEN35_SELF_SPEC_ROUTE_MEMORY_ROOT=/path` |\nDiagnostic/future-wiring hook: resolve a same-weight self-spec proposal route in `qwen35_generate` using the shared route table. Current product generation does not execute the PCA-updown self-spec runner, so hits are reported and the decode path remains unchanged. |\n`QWEN35_SELF_SPEC_ROUTE_KEY=KEY` |\nOptional caller-certified key for `QWEN35_SELF_SPEC_ROUTE_MEMORY_ROOT` ; without it, lookup uses exact prompt text and token ids. |\n`QWEN35_SELF_SPEC_UPDOWN_ADAPTERS_PATH=/path.json` |\nOptional diagnostic/future-wiring hook paired with `QWEN35_SELF_SPEC_ROUTE_MEMORY_ROOT` : validate a `qwen35_ffn_updown_adapter_v1` artifact for a selected `pca_updown` route and report validity without changing decode behavior. |\n`QWEN35_NATIVE_TOKENIZER_OFF=1` |\nDisable the native Crystal Qwen BPE encoder and use the external `llama-tokenize` bootstrap path. |\n`QWEN35_PREPARE_STATE_OFF=1` |\nDisable eager Metal state-buffer preparation in `qwen35_generate` . By default the CLI prepares KV/DeltaNet buffers before timing prompt ingest. |\n`QWEN35_METAL_PROFILE=1` |\nEnable Metal dispatch/profile attribution for the timed decode region in `qwen35_generate` . The report includes wave/group timings, prefill/source-replay phase traces, matmul logical traffic, conversion traffic, and CPU fallback counts. |\n`QWEN35_CHAT=1` |\nRender the input through the minimal Qwen 3.5/3.6 chat-template path before tokenization. |\n`QWEN35_CHAT_SYSTEM=\"...\"` |\nOptional system message used by the chat-template renderer. |\n`QWEN35_TOOLS_JSON='[...]'` |\nEnable Qwen XML-style function-calling prompt rendering and parsed `<tool_call>` output reporting. The value must be a JSON array of tool definitions. |\n`QWEN35_MESSAGES_JSON='[...]'` |\nRender an OpenAI/CrystalBall-style message array through the Qwen chat-template path. Supports `user` , `system` , `assistant` with `tool_calls` , and `tool` messages. |\n`QWEN35_TOOL_RESPONSE_JSON=simple|openai` |\nEmit a machine-readable `=== Tool response JSON ===` block after generation. `simple` matches CrystalBall's local provider shape; `openai` wraps calls as OpenAI-style function tool calls. With `QWEN35_TOOLS_JSON` , scalar arguments are normalized using the tool schema where possible. |\n`QWEN35_CONSTRAINED_LITERAL_PREFIX='...'` |\nExperimental greedy-only constrained decoding probe. Forces an exact literal prefix using tokenizer-derived allowed-token frontiers and the constrained Q6 head path, then falls back to normal greedy decode after the literal completes. Currently incompatible with prompt-cache/speculative/n-gram fast paths. |\n`QWEN35_CONSTRAINED_TOOL_CALL_PREFIX=1` |\nExperimental greedy-only structured tool-call mode. Requires `QWEN35_TOOLS_JSON` ; constrains Qwen XML `<tool_call>\\n<function=...>\\n` prefixes over the available function names, required `<parameter=...>\\n` tags in schema order, lets the model choose schema-valid optional parameter tags before final close, constrains finite enum/boolean values and small bounded integer ranges, falls back for free-form single-line values, constrains the inter-parameter and final closing tags after value newlines, stops generation after a complete constrained tool call, batches deterministic literal spans by default, and emits a `tool constraint summary` line with constrained-stage, forced-span, and free-form fallback counts. |\n`QWEN35_CONSTRAINED_FORCE_SPAN_OFF=1` |\nDisable deterministic literal-span batching inside constrained structured decoding. This is a diagnostic kill switch; the default batches grammar-proven spans with exact body-only state updates and skips intermediate lm-head ranking. |\n`QWEN35_DECODE_POLICY=greedy|ngram|speculative|mtp|auto` |\nExplicit decode-mode selector. `auto` chooses the exact fail-closed n-gram path with risk gating; explicit policy overrides legacy mode envs. `mtp` requires `QWEN35_MTP_GGUF_PATH` and remains explicit/default-off. |\n`QWEN35_TRACE_STEPS_OFF=1` |\nSuppress per-token/per-cycle trace lines in `qwen35_generate` while keeping summaries and final output. |\n`QWEN35_QUIET=1` |\nAlias for suppressing per-step traces in `qwen35_generate` ; useful for cleaner local timing. |\n`QWEN35_NGRAM_DECODE=1` |\nEnable exact n-gram speculative decode in `qwen35_generate` . |\n`QWEN35_NGRAM_GAMMA=32` |\nMaximum n-gram verifier chunk size. |\n`QWEN35_NGRAM_MIN=6` |\nMinimum repeated suffix length before n-gram drafting. |\n`QWEN35_NGRAM_MAX=8` |\nMaximum suffix length to search for n-gram drafting. |\n`QWEN35_NGRAM_MIN_CANDIDATES=N` |\nSkip n-gram proposals shorter than `N` candidates. In `auto` , the default is `8` ; explicit `ngram` keeps the old default `0` unless set. |\n`QWEN35_NGRAM_STAGE_MIN=N` |\nSplit only n-gram verifier chunks with at least `N` candidates into staged subchunks. In `auto` , the default is `QWEN35_NGRAM_GAMMA + 1` , so the common full chunk is kept intact unless overridden. Explicit `ngram` keeps the old default `0` unless set. |\n`QWEN35_NGRAM_RISK_GATE=0|1` |\nIn `auto` , the exact candidate-shape risk gate is enabled by default; set `0` to disable. Explicit `ngram` keeps the old default off unless set to `1` . |\n`QWEN35_NGRAM_RISK_MIN_SIZE=16` |\nCandidate size threshold used by the n-gram risk gate. This is independent from `QWEN35_NGRAM_STAGE_MIN` , so staging can be disabled without weakening fail-closed risk checks. |\n`QWEN35_NGRAM_CORRIDOR_GATE=0|1` |\nIn `auto` , require runtime repeat-corridor evidence for untrusted local suffix n-gram proposals. Trusted source-history cursor replay bypasses this gate after prefix validation. |\n`QWEN35_NGRAM_CORRIDOR_MATCH_LEN_MIN=8` |\nIn `auto` , allow untrusted suffix proposals when the repeated suffix match reaches this length. Explicit `ngram` defaults to `0` . |\n`QWEN35_NGRAM_CORRIDOR_LAG8_MIN=2.0` |\nIn `auto` , lag-8 repetition alone is disabled as a product certificate after template false positives; set a lower value only for A/B. Explicit `ngram` defaults to `0.5` . |\n`QWEN35_NGRAM_RECURSIVE_OFF=1` |\nDisable recursive n-gram extension through scratch history. |\n`QWEN35_NGRAM_DISABLE_AFTER_REJECT_OFF=1` |\nExploration mode: keep trying n-gram chunks after first rejection. |\n`QWEN35_NGRAM_REPLAY_ON_REJECT=1` |\nResearch/fast-path mode: skip n-gram target-state backups and rebuild the exact target state only after a non-final n-gram reject. Use with the default `auto` risk gate; it can regress badly when a large bad n-gram chunk is forced through verification. |\n`QWEN35_NGRAM_CACHE_MIN_REMAINING=64` |\nIn `auto` , require at least this many remaining requested tokens before using source-history/cache n-gram proposals. This keeps short cache-hit requests on the cheaper exact replay/greedy path. Explicit `ngram` defaults to `0` unless set. |\n`QWEN35_SPECULATIVE_DECODE=1` |\nEnable exact neural speculative decode in `qwen35_generate` using the 0.8B draft. |\n`QWEN35_DRAFT_MODEL=/path` |\nOverride the Qwen 3.5 draft GGUF used by neural speculative decode. |\n`QWEN35_SPEC_GAMMA=4` |\nInitial neural draft chunk size in `qwen35_generate` . |\n`QWEN35_SPEC_MAX_GAMMA=32` |\nMaximum adaptive neural draft chunk size. |\n`QWEN35_SPEC_PLAIN_FALLBACK_OFF=1` |\nDisable target-only fallback after low-gamma speculative rejection. Useful for A/B experiments; default fallback is faster on rejection-heavy prompts. |\n`QWEN35_SPEC_PLAIN_FALLBACK_GAMMA=2` |\nGamma threshold at or below which rejected neural speculative decode falls back to target-only generation. |\n`QWEN35_SPEC_BOOTSTRAP_GAMMA=N` |\nDefault-off neural speculative jump after a fully accepted initial chunk. Can help 100%-accept runs; may regress prompts that reject after an accepted prefix. |\n`QWEN35_SPEC_SINGLE_FAST_OFF=1` |\nDisable the exact gamma=1 accepted-token fast path in neural speculative decode. Mostly useful when target-only fallback is disabled for A/B experiments. |\n`QWEN35_SPEC_VERIFY=chunk-inplace|hybrid|serial` |\nChoose neural speculative verifier strategy. Default `chunk-inplace` is best for high-accept prompts; `hybrid` can help first-cycle partial-reject prompts. |\n`QWEN35_SPEC_SKIP_DRAFT_BEFORE_FALLBACK_OFF=1` |\nDisable the exact optimization that skips draft resync work when a rejection is guaranteed to enter target-only fallback. |\n`QWEN35_SPEC_SKIP_DRAFT_BACKUP_BEFORE_FALLBACK_OFF=1` |\nDisable the matching draft-backup skip before fallback-bound speculative chunks. |\n`QWEN35_HEAD_FULL_ROWS_GUARDED=1` |\nExperimental speculative-verifier accelerator for large accepted chunks; uses a margin guard and exact fallback for low-margin rows. |\n`QWEN35_HEAD_FULL_ROWS_MARGIN=0.25` |\nMargin threshold for the guarded full-row verifier route. Higher is safer but falls back more often. |\n`QWEN35_FFN_DOWN_ADD_FUSED_OFF=1` |\nDisable decode-wave FFN-down residual-add fusion for Q4/Q6 target and Q8 draft experiments. |\n`QWEN35_Q4K_PAIR_H16_MIN_BATCH=64` |\nTune the prefill Q4 gate/up shared H16 conversion threshold. The current default enables sharing from pp64 upward after refreshed A/B showed a small exact win. |\n`QWEN35_Q4K_H16_B48_OFF=1` |\nDisable the exact 48-token Q4_K H16 prefill tile. The default route uses this only for exact 48-token chunks. |\n`QWEN35_Q4K_H16_B64_OFF=1` |\nDisable the exact wide-batch Q4_K H16 prefill GEMM. The default route uses a 64-token batch tile for prompt chunks that are exact multiples of 64; irregular chunk sizes stay on the older 32-token tile to avoid tail regressions. |\n`QWEN35_Q4K_H16_B64_TAIL_MIN=N` |\nExperimental exact prefill probe: for non-64 prompt chunks at least `N` tokens, allow the B64 Q4_K H16 tile and its tail-safe B64 up+SwiGLU fusion on the underfilled final tile. Exact row-pack shapes with dedicated tiles stay on their proven routes. Default off because prior long-prompt tail evidence is shape-sensitive; use for A/B only. |\n`QWEN35_Q4K_H16_B80_OFF=1` |\nDisable the exact 80-token Q4_K H16 prefill tile. The default route uses this only for exact 80-token chunks. |\n`QWEN35_Q4K_H16_B96_OFF=1` |\nDisable the exact 96-token Q4_K H16 prefill tile. The default route uses this only for exact 96-token chunks; wider irregular chunks still avoid B96 after pp160/pp192 regressions. |\n`QWEN35_Q4K_H16_B112_OFF=1` |\nDisable the exact 112-token Q4_K H16 prefill tile. The default route uses this only for exact 112-token chunks. |\n`QWEN35_ADDNORM_H16_FFN=1` |\nExperimental exact prefill route: residual add+RMSNorm writes H16 rows directly for following Q4_H16 FFN gate/up GEMMs. Default off; current evidence is mixed and it is retained only as an attribution/probe knob. |\n`QWEN35_SWIGLU_H16_DOWN=1` |\nExperimental exact prefill route: SwiGLU writes H16 activations directly for FFN-down Q4/Q5/Q6 batch GEMM consumers. Default off; it removes large conversion traffic at long prompts, but paired wall timing is mixed and not a promotion signal yet. |\n`QWEN35_RMSNORM_H16_PROJ=1` |\nExperimental exact prefill route: RMSNorm writes f32 plus H16 rows so Q4/Q5/Q6 projection GEMMs can skip their separate input conversion. Default off; useful for attribution, but current paired wall timing is noise-level. |\n`QWEN35_DN_POST_H16_OPROJ=1` |\nExperimental exact prefill route: recurrent DeltaNet post-gate writes H16 rows directly for following o_proj GEMMs. Default off; completes the conversion-atlas probe set, but current wall timing is neutral. |\n`QWEN35_REC_PROJ_SHARED_H16_OFF=1` |\nDisable the exact recurrent prefill projection optimization that shares one H16 input conversion between Q5 qkv and Q4 gate GEMMs. |\n`QWEN35_PREFILL_CHUNK_OFF=1` |\nForce older non-chunked prefill path. |\n`QWEN35_DECODE_WAVE_OFF=1` |\nForce older non-wave decode path. |\n\nStructured tool-call span batching can be regression-tested with:\n\n```\nREPS=2 scripts/qwen35_structured_span_suite.sh\n```\n\nThe current Qwen API is low-level and intended for native inference experiments:\n\n```\nrequire \"ml/gguf/qwen35_cpu\"\nrequire \"ml/gguf/qwen35_weights\"\n\nmodel = \"/path/to/Qwen3.5-9B-Q4_K_M.gguf\"\nweights = ML::GGUF::Qwen35Weights.from_gguf(model)\nstate = ML::GGUF::Qwen35CPU::State.new(weights.hparams, max_seq: 1024)\n\nprompt_ids = [760_i32, 6511_i32, 314_i32, 9338_i32, 13_i32]\nnext_id, next_logit = ML::GGUF::Qwen35CPU.prefill_tokens_top1(weights, prompt_ids, 0, state)\n\n64.times do |i|\n  puts next_id\n  next_id, next_logit = ML::GGUF::Qwen35CPU.forward_top1(weights, next_id, prompt_ids.size + i, state)\nend\n```\n\nWhen linking an executable that uses Metal, include the bridge object and Apple frameworks:\n\n```\ncrystal build your_app.cr \\\n  --link-flags=\"$(pwd)/build/bridge.o -framework Metal -framework Foundation -lc++\"\n```\n\nFor CPU-only builds:\n\n```\ncrystal build -Dcpu_only your_app.cr\n```\n\nBuild the matched benchmark:\n\n```\ncrystal build --release --no-debug \\\n  --link-flags=\"$(pwd)/build/bridge.o -framework Metal -framework Foundation -lc++\" \\\n  bin/benchmark_qwen_vs_llama.cr \\\n  -o build/benchmark_qwen_vs_llama\n```\n\nRun a normal first-run prefill/decode comparison:\n\n```\n./build/benchmark_qwen_vs_llama \\\n  --model ~/.cache/lm-studio/models/lmstudio-community/Qwen3.5-9B-GGUF/Qwen3.5-9B-Q4_K_M.gguf \\\n  --llama-bench ~/SrcArchives/AI/llama.cpp/build/bin/llama-bench \\\n  --prompt=64 \\\n  --gen=64 \\\n  --reps=5 \\\n  --warmup=2\n```\n\nFor publishable measurements, wait for a quiet host:\n\n```\n./build/benchmark_qwen_vs_llama \\\n  --prompt=64 \\\n  --gen=64 \\\n  --reps=5 \\\n  --warmup=2 \\\n  --wait-quiet-ms=60000 \\\n  --require-quiet\n```\n\nAdditional benchmark modes:\n\n```\n# Default native decode now matches llama-bench `tg`: decoder body only,\n# with no output logits/head readback. Product-shaped greedy decode is:\n./build/benchmark_qwen_vs_llama --native-decode-top1\n\n# Fresh State per repetition, but Metal state buffers are prepared before\n# the timed prefill. This measures prompt ingest without first-touch buffer\n# allocation/zeroing in the timed region.\n./build/benchmark_qwen_vs_llama --native-prefill-prepare-state\n\n# State buffers allocated once, then reset between reps.\n./build/benchmark_qwen_vs_llama --native-prefill-prealloc\n\n# Exact prompt-cache restore after one seeded native prefill.\n./build/benchmark_qwen_vs_llama --native-prefill-cache\n\n# Prompt-cache prefix restore plus exact replay of the last N prompt tokens.\n./build/benchmark_qwen_vs_llama --native-prefill-cache-prefix-suffix=8\n```\n\nLatest guarded relaxed-host Qwen3.5-9B Q4_K_M body-only rows on M2 Max:\n\n| prompt/gen | cogni-ml pp | llama.cpp pp | pp gap | cogni-ml tg | llama.cpp tg | tg gap |\n|---|---|---|---|---|---|---|\n| 64/64 | 480.06 tok/s | 458.85 tok/s | +4.62% | 53.75 tok/s | 48.15 tok/s | +11.63% |\n| 256/64 | 567.60 tok/s | 566.21 tok/s | +0.25% | 53.43 tok/s | 48.26 tok/s | +10.72% |\n| 1024/64 | 582.74 tok/s | 574.77 tok/s | +1.39% | 53.14 tok/s | 48.24 tok/s | +10.16% |\n\nThese rows use `--native-prefill-prealloc`\n\n, `--threads=8`\n\n, and disabled load\nwarnings. Treat them as guarded relaxed measurements, not publishable quiet-host\nABBA evidence.\n\nFor Qwen3.6 MTP / quant baselines, first inspect the local/HF matrix:\n\n```\ncrystal build --release bin/qwen36_mtp_baseline_matrix.cr \\\n  -o build/qwen36_mtp_baseline_matrix\n\n./build/qwen36_mtp_baseline_matrix\n```\n\nThe matrix prints local-path detection plus llama.cpp command lines for the\ncurrent plain `Q4_K_M`\n\ntarget and external MTP GGUF baselines such as\n`unsloth/Qwen3.6-27B-MTP-GGUF:IQ4_NL`\n\n,\n`unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q4_K_XL`\n\n, and\n`AtomicChat/Qwen3.6-27B-UDT-MTP-GGUF:Q4_K_XL_MTP`\n\n. Use `--run-available`\n\nonly\nafter the relevant files are local and the llama.cpp binary advertises\n`draft-mtp`\n\n.\n\n`benchmark_qwen_vs_llama`\n\nalso accepts llama.cpp KV-cache options for matching\nlong-context/product configurations:\n\n```\n./build/benchmark_qwen_vs_llama \\\n  --model ~/.cache/lm-studio/models/lmstudio-community/Qwen3.6-27B-GGUF/Qwen3.6-27B-Q4_K_M.gguf \\\n  --prompt=512 \\\n  --gen=128 \\\n  --llama-cache-k=q8_0 \\\n  --llama-cache-v=q4_0\n```\n\nBoundary: native `cogni-ml`\n\ncurrently supports the K-quants used by our\nQ4_K_M/Q5_K/Q6_K/Q8_0 paths. IQ/UD MTP GGUFs are external llama.cpp baselines\nuntil native IQ/UD quant loaders are implemented.\n\nFresh local M2 Max 64GB relaxed-load snapshot after the shared-H16 recurrent projection cleanup, Qwen 3.5 9B Q4_K_M, llama.cpp `llama-bench`\n\n, `prompt=64`\n\n, `gen=64`\n\n, `reps=3`\n\n, `warmup=1`\n\n, flash-attention off:\n\n| Mode | cogni-ml | llama.cpp | Gap |\n|---|---|---|---|\n| First-run prefill | 426.70 tok/s p50 | 455.10 tok/s avg | -6.24% |\n| Fresh state, prepared Metal buffers | 449.73 tok/s p50 | 464.78 tok/s avg | -3.24% |\n| Prefill with preallocated state | 448.60 tok/s p50 | 465.91 tok/s avg | -3.71% |\n| Prompt-cache restore | 1350.65 tok/s p50 | 465.80 tok/s avg | +189.97% |\n| Plain greedy decode, first-run bench | 48.67 tok/s p50 | 46.67 tok/s avg | +4.29% |\n| Plain greedy decode, prepared-state bench | 48.59 tok/s p50 | 46.43 tok/s avg | +4.67% |\n| Plain greedy decode, preallocated bench | 48.51 tok/s p50 | 46.63 tok/s avg | +4.05% |\n| Plain greedy decode, prompt-cache bench | 48.52 tok/s p50 | 46.58 tok/s avg | +4.18% |\n\nNotes:\n\n- The table is a local engineering snapshot, not a lab-clean public benchmark.\n- First-run prefill is still behind llama.cpp on this machine. The native wins currently come from state reuse, prompt-cache restore, and exact speculative decode.\n`--native-prefill-prepare-state`\n\nuses a fresh`State`\n\nper repetition but calls`Qwen35CPU.prepare_state_metal!`\n\nbefore timing. This is useful for server-style latency where a session object can be prepared before the prompt arrives.`--native-prefill-cache`\n\nmeasures exact restore of a previously computed prompt state; it is not a first-run prefill replacement.- Short decode runs are noisy on a desktop system. The two plain decode rows above are intentionally both shown: treat plain decode as parity-to-faster, not as a stable public margin without a quiet rerun.\n\nSame-host CUDA snapshot, RTX 5060 Ti, Qwen 3.5 9B Q4_K_M, `gen=64`\n\n:\n\n| Mode | Speed | Notes |\n|---|---|---|\n| llama.cpp CUDA plain decode | 67.18 tok/s, 14.89 ms/tok | `llama-bench` `tg64` on the same host/model |\n| cogni-ml CUDA plain greedy probe | ~42.6-42.9 tok/s, ~23.3-23.5 ms/tok | full resident CUDA target path, no proposal reuse |\n| cogni-ml CUDA source/cache cursor, risk-gated | 71.76 tok/s, 13.94 ms/tok | exact output; 62/62 active-verified tokens plus two serial cursor advances |\n| cogni-ml CUDA trusted source/cache cursor | 86.61 tok/s, 11.55 ms/tok | exact output; 64/64 active-verified tokens, chunks `4,4,8,16,16,16` |\n| cogni-ml CUDA trusted bulk replay | 91.69 tok/s, 10.91 ms/tok | exact output; 64/64 active-verified tokens, one `64` -token WBA chunk |\n| cogni-ml CUDA trusted artifact restore, device-resident | restore-only lower bound: 0.189 ms / 64 cached tokens | no verifier recompute; requires validated post-span state artifact |\n| cogni-ml CUDA trusted artifact restore, host-backed | 12.259 ms / 64 cached tokens | restores a full `max_seq=128` decode state from host memory; excludes durable IO/hash lookup |\n| cogni-ml CUDA trusted artifact restore, host-backed live KV | 11.358 ms / 64 cached tokens | restores recurrent state plus live KV rows only; recurrent state dominates at short context |\n| cogni-ml CUDA trusted artifact IO/hash, live KV | write/read/hash/restore: 17.695/42.046/59.556/13.003 ms | persistent-path diagnostic for a 56.9 MB artifact; exact cached output still `64/64` |\n| cogni-ml CUDA recurrent block-INT8 artifact codec | recurrent bytes `52.7MB -> ~13.2MB` ; one-token continuation parity passed on tested cursors |\nscalar CPU encode/decode is too slow; treat as codec feasibility evidence, not production restore |\n| invalid trusted cursor | ~42.3 tok/s, ~23.64 ms/tok | fails closed: zero proposals, active verifier disabled, near plain fallback |\n\nCUDA cache-replay caveats:\n\n- The fast rows are not arbitrary generation. They measure exact replay from a validated session/source cursor, which is the intended primitive for prompt/session cache hits and repeated known spans.\n- Exact greedy parity is preserved by verifying every proposed token through the target stack before committing it.\n- Wrong cursors must fail closed. The trusted-source mode is only trusted after the source-prefix gate passes; otherwise it falls back near plain decode speed instead of accepting proposal tokens.\n- The host-backed live-KV row excludes durable cache lookup, artifact IO, and hashing; the IO/hash row includes a simple persistent-path file write/read plus host SHA-256 tool timing, but still excludes pg/session lookup and production artifact indexing.\n- Trusted artifact restore is a cache-hit fast-forward path, not speculative decoding. It is only exact if the restored state artifact and emitted token span are validated against the same model/tokenizer/config/source contract.\n\nLocal Metal resident cache snapshot, M2 Max, Qwen 3.5 9B Q4_K_M, prompt\n`alpha beta gamma delta alpha beta gamma delta`\n\n:\n\n| Mode | Speed | Notes |\n|---|---|---|\n| cogni-ml Metal plain greedy resident | `~24.97 ms/tok` |\nfresh request state, same process, no proposal reuse |\n| cogni-ml Metal exact source replay | `~4.60 ms/tok` |\nsynthetic resident prompt-state copy plus exact bulk verifier |\n| cogni-ml Metal Store source replay, resident states off | `~5.82 ms/tok` |\nreal prompt-cache Store path; cold artifact restore dominates restore phase |\n| cogni-ml Metal Store source replay, resident states on | `~4.51-4.55 ms/tok` |\nreal Store hot state path; remaining wall is verifier body |\n| cogni-ml Metal Store fast-forward, resident states off | `~1.45 ms/tok` for 64 cached tokens |\ncold artifact restore; no verifier body |\n| cogni-ml Metal Store fast-forward, resident states on | `~0.07 ms/tok` for 64 cached tokens; `~0.02 ms/tok` for 256 cached tokens |\nhot validated state restore plus cached token emission; no verifier body |\n`qwen35_generate` direct output fast-forward |\n`~1-2 ms` total for a 16-token cached span even with 5k irrelevant legacy manifest rows per cache file |\none-shot CLI hit reads a per-key output certificate, validates prompt/output/text/exact-span hashes before opening GGUF, emits cached ids/text, and exits |\n| Metal encoded BF16 artifact read+restore | `~26.35 ms` for a 28.4MB BF16 recurrent artifact in a one-prompt smoke |\ndirect encoded BF16 recurrent decode into prepared Metal buffers; avoids CPU BF16 decode path measured at `~715 ms` , but resident decoded templates remain faster at `~1.7 ms` restore-only |\n\nLatest focused resident-cache smoke, M2 Max, Qwen3.5 9B Q4_K_M, prompt14/gen8\n(`/tmp/qwen35_warm_cache_modes_20260528095435`\n\n): plain resident greedy p50\n`265.185 ms`\n\n; Store replay without resident state p50 `156.728 ms`\n\nwith p50\nrestore `44.817 ms`\n\n; Store replay with `--resident-states 2`\n\np50 `115.932 ms`\n\nwith p50 restore `3.367 ms`\n\n; state fast-forward p50 `3.657 ms`\n\n; serving-route\ncontinuation p50 `4.304 ms`\n\n; active cursor p50 `0.007 ms`\n\n; direct-output\ncertificate p50 `0.015 ms`\n\n. These are warm-process session-cache numbers, not\none-shot CLI latency.\n\nMetal cache caveat: source replay and fast-forward are different contracts. Source replay still verifies the known span through the exact target stack. Fast-forward skips that body and is only legal after validating the cached post-span state artifact and full emitted-token history. The CLI output-only hit also requires cached generated text for exactly the requested output length; it is exact for the emitted cached text, but it does not restore a continuation state because the process exits.\n\nNeural draft harness with Qwen 3.5 0.8B Q8_0:\n\n```\ncrystal build --release --no-debug \\\n  --link-flags=\"$(pwd)/build/bridge.o -framework Metal -framework Foundation -lc++\" \\\n  bin/qwen35_speculative_accept.cr \\\n  -o build/qwen35_speculative_accept\n\n./build/qwen35_speculative_accept \\\n  --target ~/.cache/lm-studio/models/lmstudio-community/Qwen3.5-9B-GGUF/Qwen3.5-9B-Q4_K_M.gguf \\\n  --draft ~/.cache/lm-studio/models/lmstudio-community/Qwen3.5-0.8B-GGUF/Qwen3.5-0.8B-Q8_0.gguf \\\n  --tokens 64 \\\n  --ngram \\\n  \"The capital of France is\"\n```\n\nCheap-proposal-only policy for repeated/template spans:\n\n```\nQWEN35_SPEC_NGRAM_MIN_CANDIDATES=8 \\\n./build/qwen35_speculative_accept \\\n  --target ~/.cache/lm-studio/models/lmstudio-community/Qwen3.6-27B-GGUF/Qwen3.6-27B-Q4_K_M.gguf \\\n  --draft ~/.cache/lm-studio/models/lmstudio-community/Qwen3.5-0.8B-GGUF/Qwen3.5-0.8B-Q8_0.gguf \\\n  --tokens 32 \\\n  --ngram \\\n  --ngram-risk-gate \\\n  --ngram-target-only \\\n  \"alpha beta gamma alpha beta gamma alpha beta gamma alpha\"\n```\n\nTarget-only n-gram speculative harness:\n\n```\ncrystal build --release --no-debug \\\n  --link-flags=\"$(pwd)/build/bridge.o -framework Metal -framework Foundation -lc++\" \\\n  bin/qwen35_ngram_speculative.cr \\\n  -o build/qwen35_ngram_speculative\n\n./build/qwen35_ngram_speculative \\\n  --tokens 64 \\\n  --gamma 32 \\\n  --min-ngram 6 \\\n  \"The capital of France is\"\n```\n\nBoth harnesses replay/check exact greedy target output by default unless their CLI explicitly says otherwise.\n\nFresh local speculative smoke, same M2 Max 64GB and Qwen 3.5 9B target:\n\n| Mode / prompt | Effective speed | Plain target | Notes |\n|---|---|---|---|\nNeural draft, `The capital of France is` |\n15.38 ms/tok, 65.01 tok/s | 21.98 ms/tok, 45.49 tok/s | 100% accepted, 64/64 candidates |\nNeural draft, `def fibonacci(n):` |\n21.06 ms/tok, 47.48 tok/s | 21.71 ms/tok, 46.07 tok/s | falls back after rejection; small but safe win |\nN-gram + neural, `The capital of France is` |\n10.10 ms/tok, 98.98 tok/s | 21.91 ms/tok, 45.64 tok/s | repeated-text path, 48/48 n-gram candidates accepted |\nExperimental guarded full-row verifier + neural, `The capital of France is` |\n14.32 ms/tok, 69.82 tok/s | 22.36 ms/tok, 44.73 tok/s | `QWEN35_HEAD_FULL_ROWS_GUARDED=1` , 0 fallback rows in this run |\nExperimental guarded full-row verifier + n-gram + neural, `The capital of France is` |\n9.20 ms/tok, 108.64 tok/s | noisy target run | `QWEN35_HEAD_FULL_ROWS_GUARDED=1` , 48/48 n-gram candidates accepted |\n\nSpeculative decode caveats:\n\n- The speculative paths are exact greedy verification paths, not approximate sampling shortcuts.\n- Neural speculative speed depends on draft acceptance. High-accept prompts are faster; rejection-heavy prompts quickly fall back to plain target decode.\n- In\n`qwen35_generate`\n\n, neural speculative decode is useful for longer high-accept generations. In a local 64-token smoke,`The capital of France is`\n\nmeasured`20.40 ms/tok`\n\ngreedy,`16.61 ms/tok`\n\nneural speculative, and`15.10 ms/tok`\n\nneural speculative with guarded full-row verification. A 32-token smoke was slower due fixed draft/verifier overhead. - N-gram speculation is a workload-specialized path for repeated/generated-template text.\n`QWEN35_DECODE_POLICY=auto`\n\nis the recommended product profile: risk-gated, runtime-corridor gated for untrusted local suffix replay,`min_candidates=8`\n\n, no neural fallback, and exact target-only fallback outside clean cheap-copy spans. - Prompt cache and source-history n-gram are request-level accelerators, so judge them by\n`total_ms`\n\n, not decode-only`ms/tok`\n\n. In a local repeated long-generation smoke, native tokenization removed the external tokenizer cost (`~560 ms -> ~0.1 ms`\n\n), tokenized-prompt cache made repeated tokenization effectively`0.0 ms`\n\n, and full-prompt cache plus source-history n-gram accepted`128/128`\n\ngenerated tokens with total wall around`1.55 s`\n\n. For short generations, the default`QWEN35_PROMPT_CACHE_FULL_HIT_MIN_GEN=64`\n\navoids a misleading path that only shifts first model work between`cache_restore_ms`\n\nand`decode_ms`\n\n. - In the research acceptance harness,\n`--ngram-target-only`\n\n/`QWEN35_SPEC_NGRAM_TARGET_ONLY=1`\n\nskips neural draft fallback after the cheap n-gram proposal source and uses exact target-only steps instead. On a 27B mixed JSONL gate it beat neural default on`5/7`\n\nprompts with paired ratio`0.909x`\n\nwhen combined with`--ngram-risk-gate`\n\n, but its average speed was near plain target decode; the real win is on clean repeated spans. - The n-gram risk gate now also catches small-period prefix overruns such as IP/YAML tails. A focused 27B probe kept clean\n`alpha beta gamma`\n\nrepeats at`~2.58x`\n\nwhile turning a YAML overrun from`0.80x`\n\ninto fail-closed target-only`1.006x`\n\n. - The productization smoke after the structured-tail fix kept clean 27B repeats fast (\n`~2.55x`\n\nover plain), made a YAML-like overrun fail closed (`1.003x`\n\n), and had`ngram_target_only_risk`\n\nbeat neural default on the tiny paired suite (`0.815x`\n\nratio). Treat this as opt-in CLI evidence, not a broad default claim. `QWEN35_NGRAM_REPLAY_ON_REJECT=1`\n\nis exact but deliberately opt-in. It removes rollback-copy overhead on high-confidence accepted n-gram chunks; on the local 27B+0.8B repeat8 harness it improved`ngram_router16_risk`\n\nfrom`30.85`\n\nto`30.21 ms/tok`\n\n. A forced no-risk YAML reject regressed from`71.53`\n\nto`95.86 ms/tok`\n\n, so this is not a broad default.- N-gram verifier chunks temporarily disable guarded full-row verification even if\n`QWEN35_HEAD_FULL_ROWS_GUARDED=1`\n\n, because partial n-gram rejection exposed a close-row guard failure during adversarial CLI testing. `QWEN35_HEAD_FULL_ROWS_GUARDED=1`\n\nis still an experimental research switch. The harness checks final output against plain greedy target output, but the route is not broad-defaulted because it relies on a full-row F16 top1 margin guard.- These numbers are effective decode throughput after prompt prefill; they do not make first-run prefill faster.\n\nThe embedding path targets `nomic-embed-text-v2-moe`\n\nwith a fully native Metal compute pipeline.\n\n```\nrequire \"ml\"\nrequire \"ml/gguf/nomic_bert\"\nrequire \"ml/gguf/metal_backend\"\nrequire \"ml/metal/compute_graph\"\n\nML::Metal::Device.init!\nmodel = ML::GGUF::NomicBertMoE.from_gguf(\"path/to/model.gguf\", ML::GGUF::MetalBackend.new)\n\nembedding = model.embed(\"Your text here\")\n```\n\nApple M2 Max, 38 GPU cores:\n\n| Tokens | Latency |\n|---|---|\n| 20 | 14 ms |\n| 94 | 16 ms |\n| 196 | 33 ms |\n| 433 | 70 ms |\n\n- simdgroup-matrix GEMM for Q5_K/Q6_K dequant+multiply.\n- Batched expert GEMM for MoE experts.\n- ComputeGraph wave scheduling with offset-aware dependency analysis.\n- Fused QKV split/RoPE, gate/softmax/top-k, scatter, and norm kernels.\n- GPU-driven dispatch where useful.\n\n| Model | Format | Status |\n|---|---|---|\n`Qwen3.5-9B` |\nGGUF Q4_K_M | Native Metal text generation path, active optimization target. |\n`Qwen3.5-0.8B` |\nGGUF Q8_0 | Native draft model path for speculative decode harnesses. |\n`Qwen3.6-27B` |\nGGUF Q4_K_M target | Planned/experimental scale-up target. |\n`nomic-embed-text-v2-moe` |\nGGUF Q5_K_M | Native Metal embedding pipeline. |\n| BERT-like encoders | GGUF | Via `NomicBertMoE` when the architecture matches. |\n| Other Llama/Qwen/Mistral-style models | GGUF | Via llama.cpp bindings. |\n\n```\n# shard.yml\ndependencies:\n  cogni-ml:\n    github: skuznetsov/cogni-ml\n    version: ~> 0.40.0\nmake build\nmake spec\n```\n\nCPU-only:\n\n```\nmake build_cpu\nmake spec_cpu\n```\n\nllama.cpp helper targets:\n\n```\nmake llama\nmake llama_env\n```\n\nThe Makefile searches common local, Homebrew, and system library locations for `libllama`\n\n. Override with `LLAMA_DIR`\n\n, `LLAMA_BUILD`\n\n, or `LLAMA_LIB_DIR`\n\nif needed.\n\n```\nrequire \"ml\"\n\nx = ML::Autograd::Variable.rand(2, 3, requires_grad: true, device: ML::Tensor::Device::CPU)\nlayer = ML::NN::Linear.new(3, 4, device: ML::Tensor::Device::CPU)\n\nout = layer.forward(x)\nloss = out.mean\nloss.backward\n\nopt = ML::Optim::Adam.new(layer.parameters)\nopt.step\nopt.zero_grad\nrequire \"ml/llm/llama\"\n\nML::LLM.init\nmodel = ML::LLM::Model.new(\"path/to/model.gguf\")\ngen = ML::LLM::Generator.new(model)\nputs gen.ask(\"What is Crystal?\", max_tokens: 100)\nML::LLM.cleanup\nrequire \"ml\"\nrequire \"ml/gguf/nomic_bert\"\nrequire \"ml/gguf/metal_backend\"\nrequire \"ml/metal/compute_graph\"\n\nML::Metal::Device.init!\nmodel = ML::GGUF::NomicBertMoE.from_gguf(\n  \"nomic-embed-text-v2-moe.Q5_K_M.gguf\",\n  ML::GGUF::MetalBackend.new\n)\n\nvec = model.embed(\"Crystal programming language\")\nputs \"dim=#{vec.size}\"\n\nvecs = model.embed_batch([\"Hello\", \"World\", \"Crystal\"])\n```\n\n| Kernel | Purpose |\n|---|---|\n`gemm_q4k.metal` |\nQ4_K GEMV/GEMM paths for Qwen. |\n`gemm_q56k.metal` |\nQ5_K/Q6_K/Q8_0 GEMV, top1, and helper kernels for Qwen. |\n`gemm_mm.metal` |\nsimdgroup-matrix GEMM for Q5_K/Q6_K and batched expert variants. |\n`gemm_simd.metal` |\nScalar SIMD GEMM fallback. |\n`ffn_qwen35.metal` |\nQwen FFN, add, RMSNorm, and activation helpers. |\n`delta_net.metal` |\nQwen 3.5 DeltaNet/recurrent kernels. |\n`fullattn_qwen35.metal` |\nQwen full-attention prefill/decode helpers. |\n`attn_decode_qwen35.metal` |\nQwen gated attention decode. |\n`attention_matmul.metal` |\nFlash-style attention matrix helpers. |\n`bert_fp16.metal` |\nNomic/BERT fused ops. |\n`nn.metal` |\nGeneral NN ops. |\n\n| Platform | GPU | CPU | Status |\n|---|---|---|---|\n| macOS Apple Silicon | Metal | Yes | Primary target. |\n| macOS Intel | Metal | Yes | Supported for general Metal paths; Qwen performance focus is Apple Silicon. |\n| Linux | Experimental CUDA probes | Yes | Use `-Dcpu_only` for GGUF/metadata and CUDA probe CLIs; full Qwen generation is still Metal-first. |\n| FreeBSD | No native Metal | Untested CPU-only | Not a primary CI target. |\n\nNVIDIA/CUDA support is currently an experimental backend-probe track, not a full decoder. The Qwen native generation path remains Metal-first.\n\n| Flag | Effect |\n|---|---|\n`-Dcpu_only` |\nDisable Metal and build pure CPU paths. |\n`-Duse_gguf` |\nEnable GGUF model loading where applicable. |\n\nMIT", "url": "https://wpnews.pro/news/crystal-ml-library-autograd-tensors-neural-networks-optimizers", "canonical_source": "https://github.com/skuznetsov/cogni-ml", "published_at": "2026-07-16 03:52:11+00:00", "updated_at": "2026-07-16 04:25:27.835478+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence", "ai-infrastructure", "ai-tools", "developer-tools"], "entities": ["Cogni-ML", "Apple Silicon", "Metal", "Qwen 3.5", "nomic-embed-text-v2-moe", "GGUF", "llama.cpp", "LM Studio"], "alternates": {"html": "https://wpnews.pro/news/crystal-ml-library-autograd-tensors-neural-networks-optimizers", "markdown": "https://wpnews.pro/news/crystal-ml-library-autograd-tensors-neural-networks-optimizers.md", "text": "https://wpnews.pro/news/crystal-ml-library-autograd-tensors-neural-networks-optimizers.txt", "jsonld": "https://wpnews.pro/news/crystal-ml-library-autograd-tensors-neural-networks-optimizers.jsonld"}}