The best Intel llm performance is unlocked with the urakozz github gist - I can’t post links but you can bing that.
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:------------|-------:|---------------:|-------------:|---------------:|---------------:|----------------:|
| Qwen3.8-27B | pp4096 | 2313.29 ± 8.87 | | 1860.31 ± 6.81 | 1771.10 ± 6.81 | 1860.31 ± 6.81 |
| Qwen3.8-27B | tg256 | 30.76 ± 0.01 | 31.67 ± 0.47 | | | |
Also 12k pp/s 100tok/s on qwen 3.6 35b a3b - although that model isn’t as capable as I’d like. I’m more than happy with daily driving 3.8 27b with hermes
It does take a bit of fiddling. I patched my vllm with some open PR’s to fix prefix caching and something else I’ve forgotten. Currently fumbling my way through enabling kv cache off - right now the choice is mtp or cache offload, and mtp wins for me.
In fact I’m so happy with the b70 that I have a second one arriving this week, to be run on a bifurcated pcie 4.0 slot - either tensor parallelism works or I run 2 independent models
If there’s any interest I’m happy to go into more detail