Exploiting Sparsity for Long Context Inference: Million Token on Commodity GPUs

Researchers propose a tunable top-k selection mechanism that reduces the cost of the forward pass in transformer language models by attending to only the most relevant tokens, enabling inference on context windows up to 1 million tokens using approximately 16GB of GPU RAM. The method achieves over 95% of model performance on common benchmarks while attending to less than 2% of input tokens, making long-context inference feasible on commodity hardware.

Computer Science Computation and Language Submitted on 10 Feb 2025 v1 https://arxiv.org/abs/2502.06766v1 , last revised 12 Feb 2025 this version, v2 Title:Exploiting Sparsity for Long Context Inference: Million Token Contexts on Commodity GPUs View PDF /pdf/2502.06766 HTML experimental https://arxiv.org/html/2502.06766v2 Abstract:There is growing demand for performing inference with hundreds of thousands of input tokens on trained transformer models. Inference at this extreme scale demands significant computational resources, hindering the application of transformers at long contexts on commodity i.e not data center scale hardware. To address the inference time costs associated with running self-attention based transformer language models on long contexts and enable their adoption on widely available hardware, we propose a tunable mechanism that reduces the cost of the forward pass by attending to only the most relevant tokens at every generation step using a top-k selection mechanism. We showcase the efficiency gains afforded by our method by performing inference on context windows up to 1M tokens using approximately 16GB of GPU RAM. Our experiments reveal that models are capable of handling the sparsity induced by the reduced number of keys and values. By attending to less than 2% of input tokens, we achieve over 95% of model performance on common benchmarks RULER, AlpacaEval, and Open LLM Leaderboard . Submission history From: Monte Hoover view email /show-email/4c7fec4d/2502.06766 Mon, 10 Feb 2025 18:47:04 UTC 7,688 KB v1 /abs/2502.06766v1 v2 Wed, 12 Feb 2025 15:55:37 UTC 7,689 KB References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .