arXiv:2611.21839v1 Announce Type: new Abstract: Speculative decoding accelerates vision-language models(VLMs) while preserving the target model's outputs, but its speedup diminishes under high concurrency serving as the system becomes more compute bounded as the batch size increases. This problem is pronounced for multimodal speculative decoding as the long range of the image tokens in the prompt can significantly increase the computational cost during draft token prediction. The block-diffusion drafters (e.g. DFlash, DSpark) can effectively speed up the draft model efficiency by predicting multiple draft tokens in one forward pass, but their speedup for VLMs under high concurrency is still modest (<9%). In order to understand this gap, we examine the information flow between different modalities from target model to draft model and identify the sparsity pattern of visual attention between the target and draft models: only small set of key-value heads in the draft model need direct visual feature attention in the context and only visual features from top level layers in the target model need to be attended in the context. We therefore propose VisHead-Spec, which selects a small, fixed set of key-value (KV) heads to attend to all image features from top level layers in the target model and restricts the remaining heads to text features. On Qwen3-VL-8B and MMSpec, VisHead-Spec-8 lets 8 of the drafter's 40 KV heads read the image while keeping accepted length within 0.3% of the block-diffusion draft model baseline (i.e. DSpark). At batch size 64, its throughput gain over autoregressive decoding increases to 16.0%, compared with 9.5% for DSpark; at batch size 1, its speedup remains comparable to DSpark.
VisHead-Spec: Head-Routed Visual Attention for Multimodal Speculative Decoding
A new arXiv paper (2611.21839v1) proposes VisHead-Spec, a head-routed visual attention method that raises speculative-decoding throughput for vision-language models under high-concurrency serving. On Qwen3-VL-8B with MMSpec, VisHead-Spec-8 lets 8 of the drafter's 40 KV heads read image features from the target model's top layers while keeping accepted length within 0.3% of the block-diffusion baseline DSpark; at batch size 64 its throughput gain over autoregressive decoding reaches 16.0% versus 9.5% for DSpark, while at batch size 1 speedup stays comparable to DSpark. The authors attribute the modest sub-9% gains of block-diffusion drafters like DFlash and DSpark to visual attention sparsity between target and draft models.
When serving VLMs with a block-diffusion drafter at high batch sizes, route only a small fixed set of KV heads to top-layer image features and keep the rest on text; 8 of 40 heads held accepted length within 0.3% and raised batch-64 gain to 16.0%.
more in playbook: Inference and self-hostingRun your AI side-project on zahid.host
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.