Speculative decoding can speed up local large language model inference by 1.5 to 2.5 times without changing output quality, according to research from Google and DeepMind. The technique uses a small d…
Google announced a method to retrofit Multi-Token Prediction onto frozen Gemini Nano v3 models, accelerating on-device inference for Pixel 9 and 10 series devices. The approach appends a lightweight T…
Researchers introduced JetSpec, a speculative decoding method that trains a causal parallel draft head on a frozen target model to draft entire speculative trees in one pass while preserving autoregre…
Speculative decoding optimizes LLM inference by using a cheap draft model to predict multiple tokens, which are then verified in a single forward pass of the target model, reducing memory-bandwidth bo…