{"slug": "accelerating-llm-inference-via-vector-index-based-output-embeddings", "title": "Accelerating LLM Inference via Vector Index Based Output Embeddings", "summary": "Researchers propose replacing dense output projections in LLMs with an HNSW-based vector index to speed up token selection, achieving up to 82% faster end-to-end decoding throughput for Gemma 3 270M on CPU while preserving generation quality under AlpacaEval evaluation.", "body_md": "arXiv:2608.27460v1 Announce Type: new\nAbstract: Large output embedding matrices create a significant memory bandwidth bottleneck during autoregressive decoding, especially for compact LLMs with large multilingual vocabularies. We reformulate the output projection followed by top-k token selection as a maximum inner product search over token embeddings and replace the dense vocabulary projection with an HNSW-based vector index. The resulting output head retrieves only a small candidate set of high-scoring tokens and can be integrated into existing decoding pipelines by scattering retrieved logits into a sparse full-vocabulary tensor. On CPU inference with Gemma 3, Llama 3.2, and Qwen 3 models, our method substantially accelerates the output projection and improves end-to-end batch-size-one decoding throughput by up to 82% for Gemma 3 270M, while preserving generation quality under AlpacaEval evaluation. These results suggest approximate retrieval is a practical alternative to dense output projections in latency-sensitive small-batch decoding.", "url": "https://wpnews.pro/news/accelerating-llm-inference-via-vector-index-based-output-embeddings", "canonical_source": "https://arxiv.org/abs/2608.27460", "published_at": "2026-08-31 04:00:00+00:00", "updated_at": "2026-08-31 04:23:58.107614+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "ai-research"], "entities": ["arXiv", "Gemma 3", "Llama 3.2", "Qwen 3", "AlpacaEval"], "alternates": {"html": "https://wpnews.pro/news/accelerating-llm-inference-via-vector-index-based-output-embeddings", "markdown": "https://wpnews.pro/news/accelerating-llm-inference-via-vector-index-based-output-embeddings.md", "text": "https://wpnews.pro/news/accelerating-llm-inference-via-vector-index-based-output-embeddings.txt", "jsonld": "https://wpnews.pro/news/accelerating-llm-inference-via-vector-index-based-output-embeddings.jsonld"}}