Self-contained GLM-5 serving stack: Just Vx kernels compiled to PTX

An experimental from-scratch GLM-MoE-DSA inference engine built on compiled Vx numerical kernels, with no vLLM, SGLang, Transformers or PyTorch in the serving path, has run the full trained GLM-5.3 mixed-bit GGUF checkpoint on CPU through all 78 base-decoder layers, generating "Let me" from a six-token abbreviated chat prefix in about 34 minutes at 32.2 GB peak RSS, matching an independent llama.cpp CPU reference. The project's latest implementation report says its full trained official comparison matches all 306 discrete checks but fails 112 of 308 frozen numerical checks, so automatic release promotion remains blocked. The GPU path compiles to PTX, but actual GPU execution and optimal performance remain unverified, and the server binds loopback by default with no authentication or TLS.

An experimental, from-scratch GLM-MoE-DSA inference engine using real compiled Vx numerical kernels . Python handles checkpoint IO, tensor storage, model orchestration, request scheduling and HTTP. The serving path does not use vLLM, SGLang, Transformers or PyTorch. An optional validation environment runs the pinned official model. NumPy supplies storage and an explicit test oracle; the vx backend never silently falls back to NumPy matrix operations. Current status: tested CPU prototype plus an explicit NVIDIA GPU kernel/backend path. The GPU path compiles to PTX; actual GPU execution and optimal performance remain unverified. See GPU implementation and run instructions https://github.com/vx-lang/glm-vx/blob/main/gpu/README.md . The full trained GLM-5.3 mixed-bit GGUF checkpoint has executed on CPU through all 78 base-decoder layers, generating “Let me” from a six-token abbreviated chat prefix. Both tokens match an independent llama.cpp CPU reference. This original baseline smoke took about 34 minutes and 32.2 GB peak RSS; it is not a complete answer or a practical-speed result. See actual model evidence https://github.com/vx-lang/glm-vx/blob/main/docs/gguf-evidence/real-chat-summary.json . The latest implementation report https://github.com/vx-lang/glm-vx/blob/main/docs/OPTIMIZATION-PROGRESS.md adds packed Vx kernels, batched prefill, prompt reuse and compiler candidate search. Its full trained official comparison matches all 306 discrete checks but fails 112 of 308 frozen numerical checks. Automatic release promotion remains blocked. The included tiny model has deterministic random weights and is for testing mechanics only. The October 4–5 engine audit https://github.com/vx-lang/glm-vx/blob/main/docs/ENGINE-AUDIT-20261004.md adds corrected DSA configuration defaults and cache identities, SIMD across independent output rows, fused packed GGUF batches with fixed-stride activation tiles, and a fused GPU index-score kernel. The new full trained two-token batched run matches the saved scalar Vx trace bit-for-bit across 1,898 checks, including both complete vocabulary outputs. This proves preservation of that baseline, not independent official parity. Serving research and prioritized roadmap https://github.com/vx-lang/glm-vx/blob/main/docs/SERVING-RESEARCH.md reviews GLM-specific provider work, papers, X discussions and their applicability to this engine. Prerequisites: Linux x86-64, Python 3.10+, C linker, working Vx v0.0.2 and LLVM 22. See Vx installation https://vxlang.org/docs/getting-started.html . python3 -m venv .venv .venv/bin/pip install -e '. test ' VXC=/path/to/vxc kernels/build.sh .venv/bin/python -m glm vx.server --tiny --backend vx On the implementation machine the isolated compiler is already available: cd /srv/nanocodex/workspace/glm-vx VXC=/srv/nanocodex/workspace/vx-toolchain/bin/vxc kernels/build.sh .venv/bin/python -m glm vx.server --tiny --backend vx curl http://127.0.0.1:8000/v1/completions \ -H 'Content-Type: application/json' \ -d '{"prompt": 1,2,3 ,"max tokens":8,"stream":true}' /health reports backend and capacity; /v1/models lists the local model. /v1/completions accepts one nonempty token-ID prompt or a string when a local --tokenizer tokenizer.json is provided. Greedy sampling is the default; finite nonnegative temperature and an optional seed support reproducible sampling. Unknown options are rejected. This is a completion API subset , not a claim of complete OpenAI chat/tool-call compatibility. SSE emits exact token IDs; decoded text is returned in the final event to avoid corrupting split UTF-8/BPE pieces. Final token ids contains the complete output, while earlier events contain one delta ID each. Clients must not concatenate the final full list again. The service binds loopback by default. It has no authentication or TLS; place a suitable gateway in front before exposing it to a network. GGUF setup and limitations https://github.com/vx-lang/glm-vx/blob/main/docs/GGUF.md describes the full GLM-5.3 IQ1 S checkpoint, memory-mapped loading, bounded expert decoding and CPU generation. The earlier tiny-model benchmarks remain synthetic; they are not trained-model performance evidence. Correctness gates and Vx performance audit https://github.com/vx-lang/glm-vx/blob/main/docs/CORRECTNESS-AND-PERFORMANCE.md records the remaining parity work and an isolated compiler optimization experiment. The implemented trace gate and current failures https://github.com/vx-lang/glm-vx/blob/main/docs/PARITY-PROGRESS.md now include full trained-weight layer/logit comparisons and independent codec checks. CPU prefill/decode handoff https://github.com/vx-lang/glm-vx/blob/main/docs/DISAGGREGATION.md runs distinct worker processes with validated cache transfer; it is not yet an HTTP worker pool or GPU speedup. GPU code https://github.com/vx-lang/glm-vx/blob/main/gpu/README.md adds coalesced F32/FP8 projection, fused residual/RMSNorm, warp/block reductions, split online-softmax compressed MLA, and a CUDA Driver API runtime. Use --backend vx-gpu for the explicitly hybrid serving adapter. The original CPU report https://github.com/vx-lang/glm-vx/blob/main/REPORT.md remains a baseline, not a GPU benchmark. Implemented optimizations and current evidence https://github.com/vx-lang/glm-vx/blob/main/docs/OPTIMIZATION-PROGRESS.md cover packed GGUF kernels, SIMD batched prefill, exact heap selection, prefix reuse, official eager validation and an explicit tensor-core GEMM path. The HTTP load benchmark and measurements https://github.com/vx-lang/glm-vx/blob/main/benchmarks/README.md track client TTFT, inter-token gaps, request latency and throughput with mixed prompt lengths. Prefill skips unused vocabulary heads, GPU MLP intermediates stay on-device, and decode-first scheduling bounds total prefill work per round. --decode-prefill-tokens default 4 trades prompt progress for smoother decode. Measured CPU streaming improvements carry a long-prompt TTFT tradeoff; no GPU throughput claim is made. - kernels/kernels.vx : float32 matvec, RMSNorm, LayerNorm, interleaved RoPE, stable softmax, SwiGLU, sigmoid router with correction bias, top-k and attention. - glm vx/model.py : dense and sparse blocks, selected routed experts plus shared experts, absorbed MLA decode, compressed request-local KV, full/shared DSA indexer selection, official expert reduction order and causal sequential decode. - glm vx/prefill.py : opt-in layer-wise batched prefill with grouped experts. - kernels/packed.vx : fused GGUF unpack-and-dot for all trained checkpoint formats. - glm vx/checkpoint.py : lazy read-only safetensors mmap with checked headers, BF16/F16/F32/FP8 E4M3FN decoding and block scales. No remote code execution. - glm vx/scheduler.py : decode-first interleaving, bounded optional batched prefill, prompt-plus-output token admission, cancellation, failure isolation and cleanup. Disconnect detection is cooperative between model forwards socket polling every 250 ms ; keep the TCP connection open until the reply. Shutdown raises an explicit timeout if a forward is still running, leaving weights open. Optional bounded prefix snapshots reuse prompt state; this is CPU serving, not a paged GPU KV allocator. - glm vx/server.py : bounded HTTP handlers, JSON completion and SSE transport. .venv/bin/python -m pytest tests kernels/test backend.py -q .venv/bin/python kernels/test kernels.py OPENBLAS NUM THREADS=1 PYTHONPATH=. .venv/bin/python scripts/benchmark.py \ --requests 2 --prompt 8 --tokens 8 --output benchmark.json The benchmark verifies matching greedy outputs first, excludes one-token warmup, then reports scheduler throughput and per-request time to first token on the tiny random model . It does not measure full-model quality or GPU throughput. See REPORT.md for the measured results and exact validation scope. --checkpoint DIRECTORY --tokenizer DIRECTORY/tokenizer.json is an experimental CPU path for a locally supplied checkpoint. It does not download weights. Config/header validation and synthetic loader tests are not full-checkpoint validation. Float32 dequantization is not numerically identical to a production FP8 activation-quantized GEMM path. Metadata and source revisions are recorded in metadata/sources.json and docs/model.md . The available machine has no detected NVIDIA GPU, about 93 GiB RAM and roughly 326 GiB disk free at initial inspection. The original FP8 checkpoint is about 756 GB and does not fit on this disk. The 216.7 GB mixed-bit GGUF checkpoint was downloaded in full and all six shard SHA256 hashes verified. No GPU hardware was rented or provisioned. To deliver an optimized production engine still requires: actual GPU validation and tuning, packed/fused GPU prefill, MTP, FP8 GEMM and quantized activation parity, paged/distributed KV sharing, tensor/expert parallel collectives, GPU topology-specific placement, trained-checkpoint logits/generation parity, and throughput/latency benchmarks against established engines on matching hardware and workloads. “Optimal” cannot be established from this CPU prototype.