Benchmarking llama.cpp Backends on Intel Panther Lake: Vulkan vs SYCL vs OpenVINO vs CPU

A benchmark of llama.cpp v0.4.1-dev (commit 83078fe) on an Intel Panther Lake laptop found the OpenVINO backend unable to run the Qwen3.6-35B-A3B-MTP-UD-Q4_K_M.gguf model, failing with a quantized-V-cache flash-attn error and a decode error (res = -3) because OpenVINO does not support the model's hybrid Attention + Mamba2 SSM + MTP architecture. The SYCL backend compiled but ran without Intel's Level Zero API after CMake reported the Level Zero loader and development headers missing, leaving device-to-device copies and memory allocation on the SYCL API and some optimization paths disabled. The test compared Vulkan, SYCL, OpenVINO and pure CPU builds on an Intel Core Ultra X7 358H CPU and integrated Intel Arc B390 GPU sharing ~32 GiB of LPDDR5 unified memory, with the 22 GiB model plus a ~10 GiB f16 KV cache exceeding the 30 GiB RAM pool.

Benchmarking llama.cpp Backends on Intel Panther Lake: Vulkan vs SYCL vs OpenVINO vs CPU I benchmarked the Qwen3.6-35B-A3B-MTP-UD-Q4 K M.gguf model a 35B Mixture-of-Experts model with 3B active parameters, 22 GiB quantized across all available llama.cpp compute backends on an Intel Panther Lake laptop. The goal: find out which backend delivers the best inference performance for large MoE models on integrated Intel GPU hardware. Hardware | Component | Spec | |---|---| | CPU | Intel Core Ultra X7 358H — 16c/16t, Panther Lake, max 4.8 GHz, AVX2 + AVX-VNNI no AVX-512 | | GPU | Intel Arc B390 integrated, Panther Lake, 12 Xe3 EUs | | RAM | ~32 GiB LPDDR5 UMA — unified memory with GPU + zram swap zstd | | Storage | 953 GiB NVMe SSD YMTC PC411 , LUKS encrypted, btrfs with zstd:1 compression | | OS | CachyOS Arch-based, rolling , kernel linux-cachyos-rc 7.13 | | Display driver | Mesa Vulkan ANV 26.2.2, Vulkan API 1.4.357 | The key architectural detail here is UMA Unified Memory Architecture . The Intel Arc B390 shares system RAM with the CPU — there's no dedicated VRAM. This means the 22 GiB model lives in the same 30 GiB pool as the KV cache and OS, making context size and cache quantization critical to avoid OOM. Software Setup - llama.cpp v0.4.1-dev commit 83078fe , cloned fresh from master - Model : Qwen3.6-35B-A3B-MTP-UD-Q4 K M.gguf — 35B MoE A3B active , 256 experts x 8, 41 blocks, hybrid Attention full attn every 4 layers + Mamba2 SSM, MTP nextn=1 - Builds : 4 separate CMake builds in isolated directories: - build-vulkan/ — -DGGML VULKAN=ON -DGGML NATIVE=ON - build-sycl/ — -DGGML SYCL=ON -DGGML SYCL F16=ON , compiled with Intel icpx - build-openvino/ — -DGGML OPENVINO=ON - build-cpu/ — -DGGML NATIVE=ON pure CPU, no GPU offload - Benchmark : llama-bench with pp512 prompt processing 512 tokens , tg128 token generation 128 tokens , KV cache q4 0, flash-attn on, 5 repetitions, 16 threads Issues Encountered 1. OpenVINO: Model Incompatibility The OpenVINO backend failed with two distinct errors: 1. quantized V cache requires flash attn to be enabled — When using q4 0 KV cache without flash-attn, OpenVINO can't create the context because it doesn't support flash attention for this model architecture. 2. test prompt: failed to decode prompt batch, res = -3 — When flash-attn is enabled which would allow quantized KV , OpenVINO still fails because it doesn't support the hybrid Attention + Mamba2 SSM + MTP architecture of Qwen3.6-35B. The root cause: OpenVINO's llama.cpp backend is primarily validated on dense models under 8B parameters. MoE routing, recurrent state models Mamba2/SSM , and multi-token prediction heads are outside its current scope. With f16 KV cache, the 22 GiB model + ~10 GiB KV cache exceeds the 30 GiB RAM limit, causing OOM. Verdict : OpenVINO is not viable for this model class. It would require significant upstream work to support MoE offloading and SSM state management. 2. SYCL: Level Zero API Missing During SYCL build configuration, CMake warned: CMake Warning: Level Zero loader or development headers not found, Level Zero API support disabled. Level Zero is Intel's low-level GPU API that provides additional performance features. The SYCL backend compiled and works without it, but operates in a degraded mode: - Device-to-device memory copies use SYCL API instead of Level Zero - Memory allocation uses SYCL API instead of VMM - Some optimization paths are disabled This means the SYCL results represent a "good but not optimal" configuration. Installing the Level Zero SDK could improve SYCL performance further. 3. OneAPI Installation Intel's oneAPI toolkit is 2.1 GiB offline installer and requires specific silent install syntax: sudo sh ./intel-oneapi-toolkit-2026.1.1.33 offline.sh \ -a -s --eula accept --install-dir /opt/intel/oneapi The --eula flag must come after -a argument passthrough , not as a top-level flag. The installer also warns about missing VTune GUI dependencies NSS, XCB, DRM, GTK3, etc. which are harmless on a headless/server setup but noisy in logs. 4. OpenCL Headers OpenVINO build initially failed with CL/cl2.hpp: No such file or directory . The fix was installing both opencl-headers and opencl-clhpp packages. Results | Backend | pp512 tok/s | tg128 tok/s | Speedup vs CPU | |---|---|---|---| | Vulkan | 624.53 +/- 7.06 | 32.75 +/- 0.09 | 6.1x pp, 1.7x tg | | SYCL | 609.36 +/- 4.65 | 30.62 +/- 0.10 | 6.0x pp, 1.6x tg | | CPU | 101.57 +/- 1.87 | 19.44 +/- 0.06 | baseline | | OpenVINO | -- | -- | N/A | Prompt Processing pp512 Vulkan and SYCL are neck-and-neck, with Vulkan holding a slight edge: - Vulkan : 624.53 tok/s best case - SYCL : 609.36 tok/s -2.4% - CPU : 101.57 tok/s -83.7% The 6x speedup over CPU confirms that GPU offload is essential for this model. The gap between Vulkan and SYCL is small ~2.5% , suggesting both backends are well-optimized for the compute-bound prompt processing phase. Token Generation tg128 The gap widens slightly in token generation: - Vulkan : 32.75 tok/s - SYCL : 30.62 tok/s -6.5% - CPU : 19.44 tok/s Token generation is memory-bandwidth bound each step reads the full model weights . On UMA, GPU and CPU share the same memory bus, so the 1.7x speedup over CPU is smaller than the 6x for prompt processing. The Vulkan backend's slightly better tg performance likely comes from more efficient memory access patterns in the Mesa ANV driver. Why Vulkan Beats SYCL Several factors contribute: 1. Mesa ANV maturity : The open-source Mesa Vulkan driver for Intel GPUs has years of optimization. SYCL's Level Zero backend, while functional, is newer. 2. Level Zero disabled : Without Level Zero, SYCL falls back to less efficient memory management. Enabling it could close the gap. 3. F16 vs native precision : SYCL was built with GGML SYCL F16=ON half-precision compute , which trades some accuracy for throughput. Vulkan uses the driver's native precision selection. 4. KHR coopmat : Both backends use cooperative matrix extensions, but Vulkan's implementation may have better-tuned tile sizes for this specific hardware. Key Takeaways 1. For Panther Lake with MoE models, use Vulkan. Zero setup, best performance, no oneAPI required. 2. SYCL is competitive but needs tuning. The Level Zero SDK would help, and future oneAPI releases may close the gap. 3. OpenVINO is not ready for MoE+SSM models. It's a capable framework for smaller dense models but lacks the operator support for modern architectures. 4. GPU offload is non-negotiable on UMA. A 6x speedup on prompt processing and 1.7x on token generation makes the difference between usable and sluggish for a 35B model. 5. Context size is the hidden bottleneck. With 30 GiB shared between model 22 GiB and KV cache, even q4 0 quantization limits context to ~35k tokens. Users who need long contexts must reduce offloaded layers or accept smaller context windows. Reproduction All builds, scripts, and results are in /home/g/Code/llama-bench/ : Run the full benchmark suite ./bench.sh Or run individual backends build-vulkan/bin/llama-bench -m models/Qwen3.6-35B-A3B-MTP-UD-Q4 K M.gguf -p 512 -n 128 -ngl 99 -t 16 -r 5 -fa on -ctk q4 0 -ctv q4 0 -o md source /opt/intel/oneapi/setvars.sh && build-sycl/bin/llama-bench -m models/Qwen3.6-35B-A3B-MTP-UD-Q4 K M.gguf -p 512 -n 128 -ngl 99 -t 16 -r 5 -fa on -ctk q4 0 -ctv q4 0 -o md build-cpu/bin/llama-bench -m models/Qwen3.6-35B-A3B-MTP-UD-Q4 K M.gguf -p 512 -n 128 -ngl 0 -t 16 -r 5 -fa on -ctk q4 0 -ctv q4 0 -o md