{"slug": "arcinferzero-level-zero-inference-engine-for-intel-arc-gpu-igpu-and-npu", "title": "Arcinferzero: level zero inference engine for Intel Arc GPU, iGPU and NPU", "summary": "A developer released arcinferzero, a level-zero inference engine for Intel Arc GPUs, integrated GPUs and NPUs, and published early multi-GPU benchmarks on 4x Intel B60 Pro cards running the Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf model. On a single GPU the engine reached 383.3-394.5 tokens/s on pp512 and 20.49 tokens/s on tg64, while two GPUs raised pp512 to 489.4-503.1 tokens/s and tg64 to 22.00 tokens/s; three GPUs reached 531.0-532.3 tokens/s on pp512 but fell to 17.29 tokens/s on tg64. The engine uses 16-lane kernels because the GPU lacks 8-lane sub-groups, enables XMX for prompt GEMMs with results checked against DP4a, and read 11321 MB of weights per token at 232.0 GB/s, or 54% of the GPU's measured 427 GB/s.", "body_md": "Hello,\n\nIntroducing a self-explanatory arcinferzero: level zero inference engine for Intel Arc GPU, iGPU and NPU\n\nOn going multi-GPU development on 4x B60 Pro with early results:\n\n(arcinferzero) C:\\work\\arcinferzero>.\\bin\\runz C:\\hf\\Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf --bench-pp 512 --n 64\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\nloaded: 64 layers, n_embd 5120, vocab 248320 | GPU buffers 12.85 GiB | W8 just-in-time | chunk 4096 | ctx 592 | load 10472 ms\n\npp512 run 0: 1335.8 ms  →  383.3 tokens/s\n\npp512 run 1: 1297.8 ms  →  394.5 tokens/s\n\npp512 run 2: 1302.1 ms  →  393.2 tokens/s\n\ntg64: 3122.8 ms  →  20.49 tokens/s\n\ntg steps: min 45.3, median 45.7, max 233.2 ms; first 8: 233.2 46.6 47.2 45.4 45.7 46.9 45.6 45.4\n\nweights read: 11321 MB per token, 232.0 GB/s (54 % of this GPU’s measured 427 GB/s)\n\n(arcinferzero) C:\\work\\arcinferzero>.\\bin\\runz C:\\hf\\Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf --bench-pp 512 --n 64 --gpus 2\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: 2 GPUs\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: GPU 0 of 2: 8 of 16 DeltaNet k-heads, 2 of 4 kv-heads, 8704 of 17408 FFN columns, 124160 of 248320 vocabulary rows\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: GPU 1 of 2: 8 of 16 DeltaNet k-heads, 2 of 4 kv-heads, 8704 of 17408 FFN columns, 124160 of 248320 vocabulary rows\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\nloaded: 2 GPUs, 64 layers, n_embd 5120, vocab 248320 | GPU buffers 6.55 GiB per GPU | W8 just-in-time | chunk 4096 | ctx 592 | load 15028 ms (GPU 0)\n\npp512 run 0: 1035.9 ms  →  494.3 tokens/s (sums: 128, 1008.9 ms waiting)\n\npp512 run 1: 1017.8 ms  →  503.1 tokens/s (sums: 128, 995.4 ms waiting)\n\npp512 run 2: 1046.1 ms  →  489.4 tokens/s (sums: 128, 1023.5 ms waiting)\n\ntg64: 2909.4 ms  →  22.00 tokens/s | steps min 27.7, median 28.0 ms | sums per token 128, 21.47 ms of each token waiting\n\nweights read: 11321 MB per token over 2 GPUs, 249.0 GB/s together\n\n(arcinferzero) C:\\work\\arcinferzero>.\\bin\\runz C:\\hf\\Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf --bench-pp 512 --n 64 --gpus 3\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: 3 GPUs\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: GPU 0 of 3: 6 of 16 DeltaNet k-heads, 2 of 4 kv-heads, 5888 of 17408 FFN columns, 82784 of 248320 vocabulary rows\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: GPU 2 of 3: 4 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 5632 of 17408 FFN columns, 82768 of 248320 vocabulary rows\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: GPU 1 of 3: 6 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 5888 of 17408 FFN columns, 82768 of 248320 vocabulary rows\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\nloaded: 3 GPUs, 64 layers, n_embd 5120, vocab 248320 | GPU buffers 4.78 GiB per GPU | W8 just-in-time | chunk 4096 | ctx 592 | load 11455 ms (GPU 0)\n\npp512 run 0: 961.8 ms  →  532.3 tokens/s (sums: 128, 935.5 ms waiting)\n\npp512 run 1: 964.2 ms  →  531.0 tokens/s (sums: 128, 941.7 ms waiting)\n\npp512 run 2: 962.7 ms  →  531.8 tokens/s (sums: 128, 938.2 ms waiting)\n\ntg64: 3702.2 ms  →  17.29 tokens/s | steps min 22.7, median 23.2 ms | sums per token 128, 33.61 ms of each token waiting\n\nweights read: 11329 MB per token over 3 GPUs, 195.8 GB/s together\n\n(arcinferzero) C:\\work\\arcinferzero>.\\bin\\runz C:\\hf\\Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf --bench-pp 512 --n 64 --gpus 4\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: 4 GPUs\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: GPU 0 of 4: 4 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 4352 of 17408 FFN columns, 62080 of 248320 vocabulary rows\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: GPU 3 of 4: 4 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 4352 of 17408 FFN columns, 62080 of 248320 vocabulary rows\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: GPU 1 of 4: 4 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 4352 of 17408 FFN columns, 62080 of 248320 vocabulary rows\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: GPU 2 of 4: 4 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 4352 of 17408 FFN columns, 62080 of 248320 vocabulary rows\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\nAbort was called at 252 line in file:\n\n(arcinferzero) C:\\work\\arcinferzero>\n\n(arcinferzero) C:\\work\\arcinferzero>\n\n(arcinferzero) C:\\work\\arcinferzero>\n\n(arcinferzero) C:\\work\\arcinferzero>\n\n(arcinferzero) C:\\work\\arcinferzero>.\\bin\\runz C:\\hf\\Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf --bench-pp 512 --n 64\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\nloaded: 64 layers, n_embd 5120, vocab 248320 | GPU buffers 12.85 GiB | W8 just-in-time | chunk 4096 | ctx 592 | load 8675 ms\n\npp512 run 0: 1316.5 ms  →  388.9 tokens/s\n\npp512 run 1: 1312.0 ms  →  390.2 tokens/s\n\npp512 run 2: 1311.1 ms  →  390.5 tokens/s\n\ntg64: 3009.2 ms  →  21.27 tokens/s\n\ntg steps: min 45.3, median 45.7, max 115.8 ms; first 8: 115.8 45.4 45.9 46.0 48.2 45.8 45.7 45.6\n\nweights read: 11321 MB per token, 240.8 GB/s (56 % of this GPU’s measured 430 GB/s)\n\n(arcinferzero) C:\\work\\arcinferzero>.\\bin\\runz C:\\hf\\Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf --bench-pp 512 --n 64 --gpus 2\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: 2 GPUs\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: GPU 0 of 2: 8 of 16 DeltaNet k-heads, 2 of 4 kv-heads, 8704 of 17408 FFN columns, 124160 of 248320 vocabulary rows\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: GPU 1 of 2: 8 of 16 DeltaNet k-heads, 2 of 4 kv-heads, 8704 of 17408 FFN columns, 124160 of 248320 vocabulary rows\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\nloaded: 2 GPUs, 64 layers, n_embd 5120, vocab 248320 | GPU buffers 6.55 GiB per GPU | W8 just-in-time | chunk 4096 | ctx 592 | load 14623 ms (GPU 0)\n\npp512 run 0: 1026.5 ms  →  498.8 tokens/s (sums: 128, 996.2 ms waiting)\n\npp512 run 1: 1030.6 ms  →  496.8 tokens/s (sums: 128, 1008.8 ms waiting)\n\npp512 run 2: 1025.7 ms  →  499.2 tokens/s (sums: 128, 1004.6 ms waiting)\n\ntg64: 1911.7 ms  →  33.48 tokens/s | steps min 27.8, median 28.2 ms | sums per token 128, 21.87 ms of each token waiting\n\nweights read: 11321 MB per token over 2 GPUs, 379.0 GB/s together\n\n(arcinferzero) C:\\work\\arcinferzero>.\\bin\\runz C:\\hf\\Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf --bench-pp 512 --n 64 --gpus 3\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: 3 GPUs\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: GPU 0 of 3: 6 of 16 DeltaNet k-heads, 2 of 4 kv-heads, 5888 of 17408 FFN columns, 82784 of 248320 vocabulary rows\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: GPU 1 of 3: 6 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 5888 of 17408 FFN columns, 82768 of 248320 vocabulary rows\n\narcinferzero: GPU 2 of 3: 4 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 5632 of 17408 FFN columns, 82768 of 248320 vocabulary rows\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\nAbort was called at 252 line in file:\n\n(arcinferzero) C:\\work\\arcinferzero>.\\bin\\runz C:\\hf\\Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf --bench-pp 512 --n 64 --gpus 4\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: 4 GPUs\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: GPU 0 of 4: 4 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 4352 of 17408 FFN columns, 62080 of 248320 vocabulary rows\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: GPU 1 of 4: 4 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 4352 of 17408 FFN columns, 62080 of 248320 vocabulary rows\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: GPU 3 of 4: 4 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 4352 of 17408 FFN columns, 62080 of 248320 vocabulary rows\n\narcinferzero: GPU 2 of 4: 4 of 16 DeltaNet k-heads, 1 of 4 kv-heads, 4352 of 17408 FFN columns, 62080 of 248320 vocabulary rows\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\nloaded: 4 GPUs, 64 layers, n_embd 5120, vocab 248320 | GPU buffers 3.40 GiB per GPU | W8 just-in-time | chunk 4096 | ctx 592 | load 9366 ms (GPU 0)\n\npp512 run 0: 962.8 ms  →  531.8 tokens/s (sums: 128, 927.4 ms waiting)\n\npp512 run 1: 947.2 ms  →  540.6 tokens/s (sums: 128, 925.6 ms waiting)\n\npp512 run 2: 946.8 ms  →  540.8 tokens/s (sums: 128, 923.0 ms waiting)\n\ntg64: 1315.1 ms  →  48.66 tokens/s | steps min 18.5, median 18.7 ms | sums per token 128, 12.72 ms of each token waiting\n\nweights read: 11329 MB per token over 4 GPUs, 551.3 GB/s together\n\n(arcinferzero) C:\\work\\arcinferzero>.\\bin\\runz C:\\hf\\Qwen3.6-35B-A3B-UD-IQ4_XS.gguf --bench-pp 512 --n 64\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\narcinferzero: memory limit 22.13 GiB (GPU share 23.33 - 1.2 for the driver), model + buffers 3.81 GiB\n\narcinferzero: 4 expert record sizes in 4 pools (records 1.47 MB x 37 layers 1.80 MB x 2 layers 2.00 MB x 1 layers 1.51 MB x 1 layers), padding 0.00 GiB of 14.70 GiB experts\n\narcinferzero: cache filled from the SSD: 15.79 GB in 4.2 s\n\nGPU memory: Windows budget 23.06 GiB, in use 1.77 GiB (allocated by the engine 18.05 GiB); RAM free 991.91 GiB\n\narcinferzero: expert cache 14.70 GiB (100 % of 14.70 GiB experts, 10496 slots), staging 244 MiB, file C:\\Users\\a.arcinferzero\\zero\\Qwen3.6-35B-A3B-UD-IQ4_XS-406eadfc.experts\n\nloaded: 40 layers, n_embd 2048, vocab 248320 | GPU buffers 18.05 GiB | W8 resident | chunk 8192 | ctx 592 | load 9636 ms\n\npp512 run 0: 290.2 ms  →  1764.5 tokens/s\n\npp512 run 1: 230.0 ms  →  2226.1 tokens/s\n\npp512 run 2: 227.2 ms  →  2253.0 tokens/s\n\ntg64: 958.3 ms  →  66.78 tokens/s\n\ntg steps: min 11.9, median 12.1, max 123.3 ms; first 8: 123.3 12.1 12.0 11.9 12.6 12.2 12.0 11.9\n\nweights read: 2423 MB per token, 161.8 GB/s (38 % of this GPU’s measured 430 GB/s)\n\n(arcinferzero) C:\\work\\arcinferzero>.\\bin\\runz C:\\hf\\Qwen3.6-35B-A3B-UD-IQ4_XS.gguf --bench-pp 512 --n 64\n\narcinferzero: no 8-lane sub-groups on this GPU: 16-lane kernels\n\narcinferzero: XMX on for the prompt GEMMs (results checked against DP4a)\n\narcinferzero: memory limit 22.13 GiB (GPU share 23.33 - 1.2 for the driver), model + buffers 3.81 GiB\n\narcinferzero: 4 expert record sizes in 4 pools (records 1.47 MB x 37 layers 1.80 MB x 2 layers 2.00 MB x 1 layers 1.51 MB x 1 layers), padding 0.00 GiB of 14.70 GiB experts\n\narcinferzero: cache filled from the SSD: 15.79 GB in 4.2 s\n\nGPU memory: Windows budget 23.06 GiB, in use 1.77 GiB (allocated by the engine 18.05 GiB); RAM free 991.84 GiB\n\narcinferzero: expert cache 14.70 GiB (100 % of 14.70 GiB experts, 10496 slots), staging 244 MiB, file C:\\Users\\a.arcinferzero\\zero\\Qwen3.6-35B-A3B-UD-IQ4_XS-406eadfc.experts\n\nloaded: 40 layers, n_embd 2048, vocab 248320 | GPU buffers 18.05 GiB | W8 resident | chunk 8192 | ctx 592 | load 8553 ms\n\npp512 run 0: 250.6 ms  →  2043.5 tokens/s\n\npp512 run 1: 240.9 ms  →  2125.0 tokens/s\n\npp512 run 2: 241.6 ms  →  2118.8 tokens/s\n\ntg64: 833.3 ms  →  76.80 tokens/s\n\ntg steps: min 11.8, median 12.0, max 66.6 ms; first 8: 66.6 12.1 12.2 17.6 11.9 12.0 12.1 11.9\n\nweights read: 2423 MB per token, 186.1 GB/s (43 % of this GPU’s measured 430 GB/s)\n\n(arcinferzero) C:\\work\\arcinferzero>\n\nThe target is reaching 80% single and multi-GPU potential.\n\nMore testers are appreciated.\n\nAlso, perhaps somebody can help on Intel eGPU support since I most likely will continue this on eGPU (Thunderbolt 4 on Acer Swift Go with Core Ultra 125H) since this is the easiest setup on my part, on Windows.", "url": "https://wpnews.pro/news/arcinferzero-level-zero-inference-engine-for-intel-arc-gpu-igpu-and-npu", "canonical_source": "https://forum.level1techs.com/t/arcinferzero-level-zero-inference-engine-for-intel-arc-gpu-igpu-and-npu/259856#post_1", "published_at": "2026-10-31 15:48:57+00:00", "updated_at": "2026-10-31 15:49:45.092306+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-chips", "ai-tools"], "entities": ["arcinferzero", "Intel Arc", "Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf", "Intel B60 Pro", "DP4a", "XMX"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/arcinferzero-level-zero-inference-engine-for-intel-arc-gpu-igpu-and-npu", "markdown": "https://wpnews.pro/news/arcinferzero-level-zero-inference-engine-for-intel-arc-gpu-igpu-and-npu.md", "text": "https://wpnews.pro/news/arcinferzero-level-zero-inference-engine-for-intel-arc-gpu-igpu-and-npu.txt", "jsonld": "https://wpnews.pro/news/arcinferzero-level-zero-inference-engine-for-intel-arc-gpu-igpu-and-npu.jsonld"}}