{"slug": "self-contained-glm-5-serving-stack-just-vx-kernels-compiled-to-ptx", "title": "Self-contained GLM-5 serving stack: Just Vx kernels compiled to PTX", "summary": "An experimental from-scratch GLM-MoE-DSA inference engine built on compiled Vx numerical kernels, with no vLLM, SGLang, Transformers or PyTorch in the serving path, has run the full trained GLM-5.3 mixed-bit GGUF checkpoint on CPU through all 78 base-decoder layers, generating \"Let me\" from a six-token abbreviated chat prefix in about 34 minutes at 32.2 GB peak RSS, matching an independent llama.cpp CPU reference. The project's latest implementation report says its full trained official comparison matches all 306 discrete checks but fails 112 of 308 frozen numerical checks, so automatic release promotion remains blocked. The GPU path compiles to PTX, but actual GPU execution and optimal performance remain unverified, and the server binds loopback by default with no authentication or TLS.", "body_md": "An experimental, from-scratch GLM-MoE-DSA inference engine using **real compiled\nVx numerical kernels**. Python handles checkpoint IO, tensor storage, model\norchestration, request scheduling and HTTP. The serving path does not use vLLM, SGLang, Transformers or\nPyTorch. An optional validation environment runs the pinned official model. NumPy supplies storage and an explicit test\noracle; the `vx` backend never silently falls back to NumPy matrix operations.\n\n**Current status: tested CPU prototype plus an explicit NVIDIA GPU kernel/backend path.**\nThe GPU path compiles to PTX; actual GPU execution and optimal performance remain\nunverified. See [GPU implementation and run instructions](https://github.com/vx-lang/glm-vx/blob/main/gpu/README.md).\nThe full trained GLM-5.3 mixed-bit GGUF checkpoint has executed on CPU through\nall 78 base-decoder layers, generating **“Let me”** from a six-token abbreviated\nchat prefix. Both tokens match an independent llama.cpp CPU reference. This\noriginal baseline smoke took about 34 minutes and 32.2 GB peak RSS; it is not a complete\nanswer or a practical-speed result. See [actual model evidence](https://github.com/vx-lang/glm-vx/blob/main/docs/gguf-evidence/real-chat-summary.json).\nThe latest [implementation report](https://github.com/vx-lang/glm-vx/blob/main/docs/OPTIMIZATION-PROGRESS.md) adds packed\nVx kernels, batched prefill, prompt reuse and compiler candidate search. Its full\ntrained official comparison matches all 306 discrete checks but fails 112 of\n308 frozen numerical checks. **Automatic release promotion remains blocked.**\nThe included tiny model has deterministic random weights and is for testing mechanics only.\n\nThe [October 4–5 engine audit](https://github.com/vx-lang/glm-vx/blob/main/docs/ENGINE-AUDIT-20261004.md) adds corrected DSA\nconfiguration defaults and cache identities, SIMD across independent output rows,\nfused packed GGUF batches with fixed-stride activation tiles, and a fused GPU\nindex-score kernel. The new full trained two-token batched run matches the saved\nscalar Vx trace bit-for-bit across 1,898 checks, including both complete vocabulary\noutputs. This proves preservation of that baseline, not independent official parity.\n\n[Serving research and prioritized roadmap](https://github.com/vx-lang/glm-vx/blob/main/docs/SERVING-RESEARCH.md) reviews GLM-specific\nprovider work, papers, X discussions and their applicability to this engine.\n\nPrerequisites: Linux x86-64, Python 3.10+, C linker, working Vx v0.0.2 and LLVM 22.\nSee [Vx installation](https://vxlang.org/docs/getting-started.html).\n\n```\npython3 -m venv .venv\n.venv/bin/pip install -e '.[test]'\nVXC=/path/to/vxc kernels/build.sh\n.venv/bin/python -m glm_vx.server --tiny --backend vx\n```\n\nOn the implementation machine the isolated compiler is already available:\n\n```\ncd /srv/nanocodex/workspace/glm-vx\nVXC=/srv/nanocodex/workspace/vx-toolchain/bin/vxc kernels/build.sh\n.venv/bin/python -m glm_vx.server --tiny --backend vx\ncurl http://127.0.0.1:8000/v1/completions \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"prompt\":[1,2,3],\"max_tokens\":8,\"stream\":true}'\n```\n\n`/health` reports backend and capacity; `/v1/models` lists the local model.\n`/v1/completions` accepts one nonempty token-ID prompt or a string when a local\n`--tokenizer tokenizer.json` is provided. Greedy sampling is the default; finite\nnonnegative temperature and an optional seed support reproducible sampling.\nUnknown options are rejected. This is a **completion API subset**, not a claim\nof complete OpenAI chat/tool-call compatibility. SSE emits exact token IDs;\ndecoded text is returned in the final event to avoid corrupting split UTF-8/BPE\npieces. Final `token_ids` contains the complete output, while earlier events\ncontain one delta ID each. Clients must not concatenate the final full list again.\n\nThe service binds loopback by default. It has no authentication or TLS; place a suitable gateway in front before exposing it to a network.\n\n[GGUF setup and limitations](https://github.com/vx-lang/glm-vx/blob/main/docs/GGUF.md) describes the full GLM-5.3 IQ1_S\ncheckpoint, memory-mapped loading, bounded expert decoding and CPU generation.\nThe earlier tiny-model benchmarks remain synthetic; they are not trained-model\nperformance evidence.\n\n[Correctness gates and Vx performance audit](https://github.com/vx-lang/glm-vx/blob/main/docs/CORRECTNESS-AND-PERFORMANCE.md)\nrecords the remaining parity work and an isolated compiler optimization experiment.\nThe [implemented trace gate and current failures](https://github.com/vx-lang/glm-vx/blob/main/docs/PARITY-PROGRESS.md) now\ninclude full trained-weight layer/logit comparisons and independent codec checks.\n[CPU prefill/decode handoff](https://github.com/vx-lang/glm-vx/blob/main/docs/DISAGGREGATION.md) runs distinct worker processes\nwith validated cache transfer; it is not yet an HTTP worker pool or GPU speedup.\n\n[GPU code](https://github.com/vx-lang/glm-vx/blob/main/gpu/README.md) adds coalesced F32/FP8 projection, fused residual/RMSNorm,\nwarp/block reductions, split online-softmax compressed MLA, and a CUDA Driver API\nruntime. Use `--backend vx-gpu` for the explicitly hybrid serving adapter.\nThe [original CPU report](https://github.com/vx-lang/glm-vx/blob/main/REPORT.md) remains a baseline, not a GPU benchmark.\n\n[Implemented optimizations and current evidence](https://github.com/vx-lang/glm-vx/blob/main/docs/OPTIMIZATION-PROGRESS.md)\ncover packed GGUF kernels, SIMD batched prefill, exact heap selection, prefix\nreuse, official eager validation and an explicit tensor-core GEMM path.\n\nThe [HTTP load benchmark and measurements](https://github.com/vx-lang/glm-vx/blob/main/benchmarks/README.md) track client\nTTFT, inter-token gaps, request latency and throughput with mixed prompt lengths.\nPrefill skips unused vocabulary heads, GPU MLP intermediates stay on-device,\nand decode-first scheduling bounds total prefill work per round.\n`--decode-prefill-tokens` (default 4) trades prompt progress for smoother decode.\nMeasured CPU streaming improvements carry a long-prompt TTFT tradeoff; no GPU\nthroughput claim is made.\n\n- `kernels/kernels.vx` : float32 matvec, RMSNorm, LayerNorm, interleaved RoPE,\nstable softmax, SwiGLU, sigmoid router with correction bias, top-k and attention.\n- `glm_vx/model.py` : dense and sparse blocks, selected routed experts plus shared\nexperts, absorbed MLA decode, compressed request-local KV, full/shared DSA\nindexer selection, official expert reduction order and causal sequential decode.\n- `glm_vx/prefill.py` : opt-in layer-wise batched prefill with grouped experts.\n- `kernels/packed.vx` : fused GGUF unpack-and-dot for all trained checkpoint formats.\n- `glm_vx/checkpoint.py` : lazy read-only safetensors mmap with checked headers,\nBF16/F16/F32/FP8 E4M3FN decoding and block scales. No remote code execution.\n- `glm_vx/scheduler.py` : decode-first interleaving, bounded optional batched prefill,\nprompt-plus-output token admission, cancellation, failure isolation and cleanup.\nDisconnect detection is cooperative between model forwards (socket polling\nevery 250 ms); keep the TCP connection open until the reply. Shutdown raises\nan explicit timeout if a forward is still running, leaving weights open.\nOptional bounded prefix snapshots reuse prompt state; this is CPU serving,\nnot a paged GPU KV allocator.\n- `glm_vx/server.py` : bounded HTTP handlers, JSON completion and SSE transport.\n\n```\n.venv/bin/python -m pytest tests kernels/test_backend.py -q\n.venv/bin/python kernels/test_kernels.py\nOPENBLAS_NUM_THREADS=1 PYTHONPATH=. .venv/bin/python scripts/benchmark.py \\\n  --requests 2 --prompt 8 --tokens 8 --output benchmark.json\n```\n\nThe benchmark verifies matching greedy outputs first, excludes one-token warmup,\nthen reports scheduler throughput and per-request time to first token on the\n**tiny random model**. It does not measure full-model quality or GPU throughput.\nSee `REPORT.md` for the measured results and exact validation scope.\n\n`--checkpoint DIRECTORY --tokenizer DIRECTORY/tokenizer.json` is an experimental\nCPU path for a locally supplied checkpoint. It does not download weights.\nConfig/header validation and synthetic loader tests are not full-checkpoint\nvalidation. Float32 dequantization is not numerically identical to a production\nFP8 activation-quantized GEMM path. Metadata and source revisions are recorded in\n`metadata/sources.json` and `docs/model.md`.\n\nThe available machine has no detected NVIDIA GPU, about 93 GiB RAM and roughly 326 GiB disk free at initial inspection. The original FP8 checkpoint is about 756 GB and does not fit on this disk. The 216.7 GB mixed-bit GGUF checkpoint was downloaded in full and all six shard SHA256 hashes verified. No GPU hardware was rented or provisioned.\n\nTo deliver an optimized production engine still requires: actual GPU validation and tuning, packed/fused GPU prefill, MTP, FP8 GEMM and quantized activation parity, paged/distributed KV sharing, tensor/expert parallel collectives, GPU topology-specific placement, trained-checkpoint logits/generation parity, and throughput/latency benchmarks against established engines on matching hardware and workloads. “Optimal” cannot be established from this CPU prototype.", "url": "https://wpnews.pro/news/self-contained-glm-5-serving-stack-just-vx-kernels-compiled-to-ptx", "canonical_source": "https://github.com/vx-lang/glm-vx", "published_at": "2026-10-20 17:09:50+00:00", "updated_at": "2026-10-20 17:16:57.449673+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "mlops", "ai-research"], "entities": ["GLM-5.3", "GLM-MoE-DSA", "Vx", "vLLM", "SGLang", "Transformers", "PyTorch", "llama.cpp"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/self-contained-glm-5-serving-stack-just-vx-kernels-compiled-to-ptx", "markdown": "https://wpnews.pro/news/self-contained-glm-5-serving-stack-just-vx-kernels-compiled-to-ptx.md", "text": "https://wpnews.pro/news/self-contained-glm-5-serving-stack-just-vx-kernels-compiled-to-ptx.txt", "jsonld": "https://wpnews.pro/news/self-contained-glm-5-serving-stack-just-vx-kernels-compiled-to-ptx.jsonld"}}