Ive spent the better part of the last week having my AI test my dual B70 system for inference. Got llama.cpp working.
Going to throw some benchmark results down below, from my results file in .md . My AI wrote inside << and >>
<<## Phase 1: SYCL vs Vulkan Baselines
### Methodology
Every model was tested on both backends with identical settings (q8_0/q8_0 KV cache, flash attention ON). Three metrics measured per run: prompt processing speed at 512 tokens (pp512), long prompt processing at 4096 tokens (pp4096), and generation throughput for 128 tokens (tg128).
Results
Model | Backend | tg128 tok/s | pp512 tok/s | pp4096 tok/s |
|---|---|---|---|---|
| Qwen3.6-27B-Q4_K_S | SYCL | 25.3 |
751 | 981 | | Qwen3.6-27B-Q4_K_S | Vulkan | 18.9 | 483 | 341 | | Qwen3.6-35B-MXFP4_MOE | SYCL | 52.3 | 929 | 1610 | | Qwen3.6-35B-MXFP4_MOE | Vulkan | 33.5 | 1185* | 1018 | | gemma-26B-MoE | SYCL | 59.6 | 1502 | 2139 | | gemma-26B-MoE | Vulkan | 47.7 | 910 | 275* | | gemma-31B-dense | SYCL | 23.5 | 570 | 736 | | gemma-31B-dense | Vulkan | 15.7 | 254 | 68* |
*Anomaly: Vulkan pp512 for Qwen35B and pp4096 collapse for gemma models likely caused by VRAM pressure triggering CPU fallback.
### Breakdown
SYCL dominates across the board. The tg128 gap is consistent at 1.3–1.6× faster on every model. Vulkan has one anomalous win (Qwen35B pp512) but collapses catastrophically on long prompts for gemma models — pp4096 drops to 275 tok/s on gemma-26B and 68 tok/s on gemma-31B, compared to SYCL’s 2139 and 736 respectively.
Verdict: Always use SYCL. Vulkan is not viable for production inference on Arc B70.
## Phase 2: KV Cache Quantization (f16/f16 vs q4_0/q4_0)
### Methodology
All tests ran on SYCL with flash attention ON, batch/ubatch 2048/2048. Three cache types tested: f16/f16 (full precision), q4_0/q4_0 (aggressive quantization), and q8_0/f16 (mixed).
Results — tg128 Generation Throughput
Model | f16/f16 tok/s | q4_0/q4_0 tok/s | Δ |
|---|---|---|---|
| Qwen3.6-27B-Q4_K_S | 25.7 |
25.3 | +1.4% (f16) |
| gemma-26B-MoE | 65.8 |
59.3 | +11.0% (f16) |
| gemma-31B-dense | 25.4 |
23.4 | +8.5% (f16) |
Results — pp512 Prompt Processing
Model | f16/f16 tok/s | q4_0/q4_0 tok/s | Δ |
|---|---|---|---|
| Qwen3.6-27B-Q4_K_S | 781 |
752 | +3.9% (f16) |
| gemma-26B-MoE | 1706 |
1495 | +14.1% (f16) |
| gemma-31B-dense | 707 |
569 | +24.2% (f16) |
Results — pp4096 Long Prompt Processing
Model | f16/f16 tok/s | q4_0/q4_0 tok/s | Δ |
|---|---|---|---|
| Qwen3.6-27B-Q4_K_S | 987 |
982 | +0.5% (f16) |
| gemma-26B-MoE | 2182 |
2147 | +1.6% (f16) |
| gemma-31B-dense | 755 |
734 | +2.9% (f16) |
### Breakdown
f16/f16 KV cache is consistently faster — up to +24% on pp512 for gemma-31B-dense and +11% on tg128 for gemma-26B-MoE. The benefit is most pronounced during prompt processing (where the cache is actively written) rather than generation (where it’s mostly read).
The tradeoff: f16/f16 uses roughly 4× more VRAM per cached token than q4_0/q4_0. On a 32 GB GPU, that matters for long-context workloads — but the speed gain is real and measurable.
Verdict: Use f16/f16 KV cache when VRAM allows. The +1–24% speed boost across all metrics is worth the memory cost on most models.
## Phase 3: Flash Attention ON vs OFF
### Methodology
All tests ran on SYCL with f16/f16 KV cache, batch/ubatch 2048/2048. Only flash attention toggled between runs.
### Results — tg128 Generation Throughput
Model | FA ON tok/s | FA OFF tok/s | Δ |
|---|---|---|---|
| Qwen3.6-27B-Q4_K_S | 25.6 | 25.9 |
+1.2% (FA Off) |
| Qwen3.6-35B-MXFP4_MOE | 52.5 | 53.5 |
+1.9% (FA Off) |
| gemma-26B-MoE | 65.8 |
63.4 | -3.7% (FA On wins) |
| gemma-31B-dense | 25.4 |
24.8 | -2.4% (FA On wins) |
### Breakdown
Flash attention has negligible impact on tg128 — within ±4%. Interestingly, Qwen models are slightly faster with FA off (possibly less overhead for short generation), while gemma models prefer FA on. Since flash attention saves VRAM and helps pp4096 performance, keeping it ON as default makes sense despite the marginal tg128 tradeoff.
Verdict: Keep flash attention ON. The ±4% impact on generation is noise; the VRAM savings and long-prompt benefits outweigh any micro-optimization.
## Phase 5: Concurrency Saturation Curves
### Methodology
We tested streams 1 through 20 (with fine-grained testing at 2, 3, and 4) on all four models. Each stream fires a concurrent request with max_tokens=256 and temperature 0.0. Aggregate throughput is the sum of all streams’ tok/s; per-stream tok/s shows individual degradation.
### Full Results
Model | Stream 1 | Stream 2 | Stream 3 | Stream 4 | Stream 5 | Stream 10 | Stream 15 | Stream 20 | Ceiling |
|---|---|---|---|---|---|---|---|---|---|
| gemma-4-26B-A4B-it-QAT-Q4_0 | 52.5 | 75.3 | 89.6 | 100.0 |
85.8 | 94.6 | 97.8 | 98.8 | ~99 tok/s | | gemma-4-12B-it-QAT-Q4_0 | 52.6 | 73.1 | 94.6 | 116.3 | 90.0 | 92.2 | 97.2 | 100.7 | ~101 tok/s | | Qwen3.6-35B-A3B-MXFP4_MOE | 39.6 | 48.3 | 61.4 | 72.9 | 69.2 | 66.2 | 69.7 | 71.6 | ~72 tok/s | | Qwen3.6-27B-Q8_0 | 12.7 | 20.4 | 27.4 | 32.4 | 24.6 | 28.9 | 31.8 | 33.1 | ~34 tok/s |
>>
From my other chats with google AI, intel cards seems to have 2 compute pathways or something, so it makes sense for 2 or 4 concurrency streams to have the best numbers (2 cards?), and it looks like scheduling bandwidth hits hard at 5 streams
I completely failed, or just got junk data with MTP results. LM studio shows MTP working, I must be passing the wrong flags or something, my AI built the llama.cpp handler…
Ive also been struggling to get testing done on both GPU’s, no matter what I tell my bot to do, those models end up on a single GPU. I want to try to get it across both so I can see what performance degradation/uplift there is between single and dual layer split and dual tensor split.
I think theres also a slowdown because I used Q8 for the 27b model, I read somewhere that q8 is a specific slowdown until something gets an update.