havent asked it about tianenmen square or the uyghurs yet though
honestly my company pays for deepseek and glm 5-3 flash models through API and this feels right up there, even the token rate is similar
Hate to say it, but the speed and performance difference between llamaserver and vLLM is kind of nuts, its way more intelligent with prompt caching etc, i’ve barely had to wait a second even on large prompts
Ok you know what? stretch goal:
Lets see if i can make an mxfp4 quant from the orcarouter version and mash it into the clavviger engine: https://www.youtube.com/watch?v=hFDcoX7s6rE
OK well its doing ….something, i have a feeling this will take a while
[AA] first layer took 9m55s; estimating 7h56m for 48 layers → finishes ~00:38:46
[AA] layer 0: 1539 modules (1516 with stats), 2 stage(s), capture 100.6s recapture 85.8s quant 234.8s fwd#2 74.9s io-wait 0.0s, E-moved 0, micro-batch 4
[AA] PLE table: 128 mmap’d shards x 2500012 rows (torch.bfloat16), scale no, gathered on CPU
See you gents tommorow
Stretchgoal accomplished XD, requanting isnt as hard as i though performance is near identical but now i can ask it about the uyghurs
I’m going to make a version of the qwen 3.8 27B next on mxfp4 using this engine too, i feel like i can finally use these cards to their true potential all of a sudden XD
I think this means i did it good
If you install the ROCm Validation Suite, you can use the TransferBench tool to test the p2p speed GPU to GPU and GPU to CPU. I didn’t have any issues getting p2p to work with Proxmox, but the speed was a lot lower than when running the same tests bare-metal.
Ah i know transferbench, thats how i know its still root-complex routing rather than the PCIE switch, now that VLLM and MXFP4 are sorted its my next target to fix
1 Like