Hello, Introducing a self-explanatory arcinferzero: level zero inference engine for Intel Arc GPU, iGPU and NPU
On going multi-GPU development on 4x B60 Pro with early results:
(arcinferzero) C:\work\arcinferzero>.\bin\runz C:\hf\Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf --bench-pp 512 --n 64
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
loaded: 64 layers, n_embd 5120, vocab 248320 | GPU buffers 12.85 GiB | W8 just-in-time | chunk 4096 | ctx 592 | load 10472 ms pp512 run 0: 1335.8 ms → 383.3 tokens/s
pp512 run 1: 1297.8 ms → 394.5 tokens/s
pp512 run 2: 1302.1 ms → 393.2 tokens/s
tg64: 3122.8 ms → 20.49 tokens/s tg steps: min 45.3, median 45.7, max 233.2 ms; first 8: 233.2 46.6 47.2 45.4 45.7 46.9 45.6 45.4
weights read: 11321 MB per token, 232.0 GB/s (54 % of this GPU’s measured 427 GB/s)
(arcinferzero) C:\work\arcinferzero>.\bin\runz C:\hf\Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf --bench-pp 512 --n 64 --gpus 2
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels
arcinferzero: 2 GPUs
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels
arcinferzero: GPU 0 of 2: 8 of 16 DeltaNet k-heads, 2 of 4 kv-heads, 8704 of 17408 FFN columns, 124160 of 248320 vocabulary rows
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels arcinferzero: GPU 1 of 2: 8 of 16 DeltaNet k-heads, 2 of 4 kv-heads, 8704 of 17408 FFN columns, 124160 of 248320 vocabulary rows
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
loaded: 2 GPUs, 64 layers, n_embd 5120, vocab 248320 | GPU buffers 6.55 GiB per GPU | W8 just-in-time | chunk 4096 | ctx 592 | load 15028 ms (GPU 0)
pp512 run 0: 1035.9 ms → 494.3 tokens/s (sums: 128, 1008.9 ms waiting)
pp512 run 1: 1017.8 ms → 503.1 tokens/s (sums: 128, 995.4 ms waiting)
pp512 run 2: 1046.1 ms → 489.4 tokens/s (sums: 128, 1023.5 ms waiting)
tg64: 2909.4 ms → 22.00 tokens/s | steps min 27.7, median 28.0 ms | sums per token 128, 21.47 ms of each token waiting
weights read: 11321 MB per token over 2 GPUs, 249.0 GB/s together
(arcinferzero) C:\work\arcinferzero>.\bin\runz C:\hf\Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf --bench-pp 512 --n 64 --gpus 3
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels
arcinferzero: 3 GPUs
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels
arcinferzero: GPU 0 of 3: 6 of 16 DeltaNet k-heads, 2 of 4 kv-heads, 5888 of 17408 FFN columns, 82784 of 248320 vocabulary rows
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels arcinferzero: GPU 2 of 3: 4 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 5632 of 17408 FFN columns, 82768 of 248320 vocabulary rows
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels arcinferzero: GPU 1 of 3: 6 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 5888 of 17408 FFN columns, 82768 of 248320 vocabulary rows
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
loaded: 3 GPUs, 64 layers, n_embd 5120, vocab 248320 | GPU buffers 4.78 GiB per GPU | W8 just-in-time | chunk 4096 | ctx 592 | load 11455 ms (GPU 0)
pp512 run 0: 961.8 ms → 532.3 tokens/s (sums: 128, 935.5 ms waiting)
pp512 run 1: 964.2 ms → 531.0 tokens/s (sums: 128, 941.7 ms waiting)
pp512 run 2: 962.7 ms → 531.8 tokens/s (sums: 128, 938.2 ms waiting)
tg64: 3702.2 ms → 17.29 tokens/s | steps min 22.7, median 23.2 ms | sums per token 128, 33.61 ms of each token waiting
weights read: 11329 MB per token over 3 GPUs, 195.8 GB/s together
(arcinferzero) C:\work\arcinferzero>.\bin\runz C:\hf\Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf --bench-pp 512 --n 64 --gpus 4
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels
arcinferzero: 4 GPUs
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels
arcinferzero: GPU 0 of 4: 4 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 4352 of 17408 FFN columns, 62080 of 248320 vocabulary rows
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels arcinferzero: GPU 3 of 4: 4 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 4352 of 17408 FFN columns, 62080 of 248320 vocabulary rows
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels arcinferzero: GPU 1 of 4: 4 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 4352 of 17408 FFN columns, 62080 of 248320 vocabulary rows
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels arcinferzero: GPU 2 of 4: 4 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 4352 of 17408 FFN columns, 62080 of 248320 vocabulary rows
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
Abort was called at 252 line in file:
(arcinferzero) C:\work\arcinferzero>
(arcinferzero) C:\work\arcinferzero>
(arcinferzero) C:\work\arcinferzero>
(arcinferzero) C:\work\arcinferzero>
(arcinferzero) C:\work\arcinferzero>.\bin\runz C:\hf\Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf --bench-pp 512 --n 64
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
loaded: 64 layers, n_embd 5120, vocab 248320 | GPU buffers 12.85 GiB | W8 just-in-time | chunk 4096 | ctx 592 | load 8675 ms pp512 run 0: 1316.5 ms → 388.9 tokens/s
pp512 run 1: 1312.0 ms → 390.2 tokens/s
pp512 run 2: 1311.1 ms → 390.5 tokens/s
tg64: 3009.2 ms → 21.27 tokens/s tg steps: min 45.3, median 45.7, max 115.8 ms; first 8: 115.8 45.4 45.9 46.0 48.2 45.8 45.7 45.6
weights read: 11321 MB per token, 240.8 GB/s (56 % of this GPU’s measured 430 GB/s)
(arcinferzero) C:\work\arcinferzero>.\bin\runz C:\hf\Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf --bench-pp 512 --n 64 --gpus 2
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels
arcinferzero: 2 GPUs
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels
arcinferzero: GPU 0 of 2: 8 of 16 DeltaNet k-heads, 2 of 4 kv-heads, 8704 of 17408 FFN columns, 124160 of 248320 vocabulary rows
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels arcinferzero: GPU 1 of 2: 8 of 16 DeltaNet k-heads, 2 of 4 kv-heads, 8704 of 17408 FFN columns, 124160 of 248320 vocabulary rows
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
loaded: 2 GPUs, 64 layers, n_embd 5120, vocab 248320 | GPU buffers 6.55 GiB per GPU | W8 just-in-time | chunk 4096 | ctx 592 | load 14623 ms (GPU 0)
pp512 run 0: 1026.5 ms → 498.8 tokens/s (sums: 128, 996.2 ms waiting)
pp512 run 1: 1030.6 ms → 496.8 tokens/s (sums: 128, 1008.8 ms waiting)
pp512 run 2: 1025.7 ms → 499.2 tokens/s (sums: 128, 1004.6 ms waiting)
tg64: 1911.7 ms → 33.48 tokens/s | steps min 27.8, median 28.2 ms | sums per token 128, 21.87 ms of each token waiting
weights read: 11321 MB per token over 2 GPUs, 379.0 GB/s together
(arcinferzero) C:\work\arcinferzero>.\bin\runz C:\hf\Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf --bench-pp 512 --n 64 --gpus 3
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels
arcinferzero: 3 GPUs
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels
arcinferzero: GPU 0 of 3: 6 of 16 DeltaNet k-heads, 2 of 4 kv-heads, 5888 of 17408 FFN columns, 82784 of 248320 vocabulary rows
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels
arcinferzero: GPU 1 of 3: 6 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 5888 of 17408 FFN columns, 82768 of 248320 vocabulary rows
arcinferzero: GPU 2 of 3: 4 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 5632 of 17408 FFN columns, 82768 of 248320 vocabulary rows
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
Abort was called at 252 line in file:
(arcinferzero) C:\work\arcinferzero>.\bin\runz C:\hf\Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf --bench-pp 512 --n 64 --gpus 4
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels
arcinferzero: 4 GPUs
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels
arcinferzero: GPU 0 of 4: 4 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 4352 of 17408 FFN columns, 62080 of 248320 vocabulary rows
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels
arcinferzero: GPU 1 of 4: 4 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 4352 of 17408 FFN columns, 62080 of 248320 vocabulary rows
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels arcinferzero: GPU 3 of 4: 4 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 4352 of 17408 FFN columns, 62080 of 248320 vocabulary rows
arcinferzero: GPU 2 of 4: 4 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 4352 of 17408 FFN columns, 62080 of 248320 vocabulary rows
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
loaded: 4 GPUs, 64 layers, n_embd 5120, vocab 248320 | GPU buffers 3.40 GiB per GPU | W8 just-in-time | chunk 4096 | ctx 592 | load 9366 ms (GPU 0)
pp512 run 0: 962.8 ms → 531.8 tokens/s (sums: 128, 927.4 ms waiting)
pp512 run 1: 947.2 ms → 540.6 tokens/s (sums: 128, 925.6 ms waiting)
pp512 run 2: 946.8 ms → 540.8 tokens/s (sums: 128, 923.0 ms waiting)
tg64: 1315.1 ms → 48.66 tokens/s | steps min 18.5, median 18.7 ms | sums per token 128, 12.72 ms of each token waiting
weights read: 11329 MB per token over 4 GPUs, 551.3 GB/s together
(arcinferzero) C:\work\arcinferzero>.\bin\runz C:\hf\Qwen3.6-35B-A3B-UD-IQ4_XS.gguf --bench-pp 512 --n 64
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
arcinferzero: memory limit 22.13 GiB (GPU share 23.33 - 1.2 for the driver), model + buffers 3.81 GiB
arcinferzero: 4 expert record sizes in 4 pools (records 1.47 MB x 37 layers 1.80 MB x 2 layers 2.00 MB x 1 layers 1.51 MB x 1 layers), padding 0.00 GiB of 14.70 GiB experts arcinferzero: cache filled from the SSD: 15.79 GB in 4.2 s
GPU memory: Windows budget 23.06 GiB, in use 1.77 GiB (allocated by the engine 18.05 GiB); RAM free 991.91 GiB
arcinferzero: expert cache 14.70 GiB (100 % of 14.70 GiB experts, 10496 slots), staging 244 MiB, file C:\Users\a.arcinferzero\zero\Qwen3.6-35B-A3B-UD-IQ4_XS-406eadfc.experts
loaded: 40 layers, n_embd 2048, vocab 248320 | GPU buffers 18.05 GiB | W8 resident | chunk 8192 | ctx 592 | load 9636 ms
pp512 run 0: 290.2 ms → 1764.5 tokens/s
pp512 run 1: 230.0 ms → 2226.1 tokens/s
pp512 run 2: 227.2 ms → 2253.0 tokens/s
tg64: 958.3 ms → 66.78 tokens/s tg steps: min 11.9, median 12.1, max 123.3 ms; first 8: 123.3 12.1 12.0 11.9 12.6 12.2 12.0 11.9
weights read: 2423 MB per token, 161.8 GB/s (38 % of this GPU’s measured 430 GB/s)
(arcinferzero) C:\work\arcinferzero>.\bin\runz C:\hf\Qwen3.6-35B-A3B-UD-IQ4_XS.gguf --bench-pp 512 --n 64
arcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels
arcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)
arcinferzero: memory limit 22.13 GiB (GPU share 23.33 - 1.2 for the driver), model + buffers 3.81 GiB
arcinferzero: 4 expert record sizes in 4 pools (records 1.47 MB x 37 layers 1.80 MB x 2 layers 2.00 MB x 1 layers 1.51 MB x 1 layers), padding 0.00 GiB of 14.70 GiB experts arcinferzero: cache filled from the SSD: 15.79 GB in 4.2 s
GPU memory: Windows budget 23.06 GiB, in use 1.77 GiB (allocated by the engine 18.05 GiB); RAM free 991.84 GiB
arcinferzero: expert cache 14.70 GiB (100 % of 14.70 GiB experts, 10496 slots), staging 244 MiB, file C:\Users\a.arcinferzero\zero\Qwen3.6-35B-A3B-UD-IQ4_XS-406eadfc.experts
loaded: 40 layers, n_embd 2048, vocab 248320 | GPU buffers 18.05 GiB | W8 resident | chunk 8192 | ctx 592 | load 8553 ms
pp512 run 0: 250.6 ms → 2043.5 tokens/s
pp512 run 1: 240.9 ms → 2125.0 tokens/s
pp512 run 2: 241.6 ms → 2118.8 tokens/s
tg64: 833.3 ms → 76.80 tokens/s tg steps: min 11.8, median 12.0, max 66.6 ms; first 8: 66.6 12.1 12.2 17.6 11.9 12.0 12.1 11.9
weights read: 2423 MB per token, 186.1 GB/s (43 % of this GPU’s measured 430 GB/s)
(arcinferzero) C:\work\arcinferzero> The target is reaching 80% single and multi-GPU potential.
More testers are appreciated.
Also, perhaps somebody can help on Intel eGPU support since I most likely will continue this on eGPU (Thunderbolt 4 on Acer Swift Go with Core Ultra 125H) since this is the easiest setup on my part, on Windows.