I have tried llama.cpp and vllm but so far only getting 4-6 output tokens/seconds with granite-4.2-30b-fp8
Are there any known tips to get faster output? different models, options, bespoke patches?
1 Like
lambda 2
Pick a different model? Ideally MoE.
Check out Qwen 3.8 Flash Next -
2 Likes
Thank you, I was able to get 40 tokens/seconds with this