Both of the quanters, Ubergarm and Aesedai, (who I’m eternally greatful for) have not produced a GLM5.2 quant as of yet, so I tried it myself. This took me about a week to do with only 512GB of RAM, following the instructions Ubergarm had on his GLM 5.1 quants. I couldn’t figure out a way to get my GPUS involved, some error messages kept popping up, might have just been incompetence on my part. Generating the perplexity numbers will once again take me days, but they are coming once I can spare time on the weekend. Hopefully Ubergarm you can return to doing the quants, this took me a very long time, even though it was fun to do.
I’m running with 2x56 core Emerald Rapids Xeons, 512GB of DDR5 ECC Ram @ 5600, and a single RTX 6000, and I’m getting about 15T/s @ 16k context, about 400 T/s prompt processing. The optimizations in ik_llama over the past few days for glm 5.2 seem to be really good.
/usr/bin/numactl --interleave=all /localzfs/base/exp10/ik_llama.cpp/build/bin/llama-server --alias GLM-5.2 --model /rulers/GLM5.2-GGUF/smol-IQ4_K/GLM-5.2-smol-IQ4_K-00001-of-00033.gguf --port 8080 --ctx-size 120000 -ctk q8_0 -ctv q8_0 -amb 512 -ub 4096 -b 4096 --n-gpu-layers 99 -ot "blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14)\.ffn_(gate|up|down)_exps.*=CUDA0" -ot exps=CPU --parallel 1 --threads 112 --host 0.0.0.0 --temp 1 --top-p 0.95 --top-k 40 --no-mmap --numa distribute --jinja
I discovered yesterday that the option -op 27,0,28,0,30,0,31,0
, which originally was speeding me up, is no longer. With this command line absent, almost all of the prompt preprocessing is now happening on my gpu. I’ve tried mtp and it is not speeding me up at all, which is interesting.
I and many others here took an early gamble on this stuff back in 2025, I remember jumping on the ebay emerald rapids thinking ktransformers was going to be the thing I was running. My only regret is going with a 2 socket system, maybe at some point I try to transition to a single socket 16 Dimm board of some kind, if such a thing exists. The numa nodes are terrible for memory bandwidth and multiple gpus. But at this point in 2026, what I got is what I got for now haha.
3 Likes
I’ve redone the quants to be based off of an imatrix computed against the ubergarm text corpus instead of wiki-text-raw. Turns out I didn’t know what I was doing haha.
How about running deepseek v4 from your phone with no gpu?
3 Likes
Lol what software is this?
That one is kimi k3 in C, but there are alot of others like it. This one is less the a meg of code in C99 running it no extra stack.