Here’s a datapoint from my 5090; I’ve been running qwen3.8 with GitHub - Neroued/ninfer: High-performance single-GPU inference for selected model checkpoints and GPUs. · GitHub . Similar to you some of this local model stuff has outpaced me since I last checked in, don’t completely grasp the impact of MTP and KVcache quant yet. But this ninfer seemed to have some optimizations for single gpu 5090 arch, but it may not work with the finetunes you need. I also pulled one of the uncensored models with lmstudio ran with llama-benchy as a comparison.
specs
9950x3d
5090
96GB 6400mhz
linux
| Model / Runtime | Quant | MTP Max | Decode | Prompt Eval | Approx. VRAM | Relative Speed |
|---|---|---|---|---|---|---|
| JonathanColetti/Qwen3.8-27B-Uncensored-GGUF via LM Studio/llama.cpp | Q6_K | 2 | 72.03 ± 5.66 tok/s | 2,430.18 ± 24.45 tok/s | ~28.5 GB total GPU use | 1.00× |
| neroued/Qwen3.8-27B-nvfp4-NInfer via NInfer | mixed NVFP4/FP8 | 3 | 124.08 ± 10.56 tok/s | 8,143.82 ± 52.68 tok/s | ~28.5 GB total GPU use | 1.72× |
Looks like these models tok/s can vary pretty heavily based on the type of output being generated. So, hard to say how useful this will be for you without comparing identical prompts. I’m interested in your usecase though, uncensored model + red teaming on 5090. Let me know if you find some optimal config with vLLM