A user self-hosting large language models on an AMD Radeon RX 9700 reports achieving only ~22 tokens/s with Qwen 3.8 Q5_K_m, despite fitting 132k Q8_0 context, and seeks tips for improving performance…
A developer benchmarked the Qwen3.8-27B model with NVFP4 quantization and multi-token prediction (MTP) on an RTX 5090, comparing tier configurations and sweeping spec-draft-n values at 64K context. Th…
A developer reports that speculative decoding techniques MTP, EAGLE-3, and DFlash can significantly speed up local inference of large language models in llama-cpp. Testing on an NVidia 3060 RTX 12 GB …
Google released new Gemma 4 checkpoints optimized with Quantization-Aware Training (QAT) to reduce model memory footprint for local deployment on edge devices and consumer GPUs. The QAT process minimi…
Speculative decoding optimizes LLM inference by using a cheap draft model to predict multiple tokens, which are then verified in a single forward pass of the target model, reducing memory-bandwidth bo…
A developer discovered that standard quantization pipelines for large language models silently discard multi-token prediction (MTP) heads, causing speculative decoding speedups to vanish despite the b…