Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

Researchers introduced Sarathi-Serve, an LLM inference scheduler that uses chunked-prefills and stall-free scheduling to improve throughput-latency tradeoffs, achieving 2.6x higher serving capacity for Mistral-7B on a single A100 GPU and up to 3.7x for Yi-34B on two A100 GPUs compared to vLLM, with up to 5.6x gain for Falcon-180B with pipeline parallelism. The source code is available on GitHub.

Computer Science Machine Learning Submitted on 4 Mar 2024 v1 https://arxiv.org/abs/2403.02310v1 , last revised 17 Jun 2024 this version, v3 Title:Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve View PDF /pdf/2403.02310 HTML experimental https://arxiv.org/html/2403.02310v3 Abstract:Each LLM serving request goes through two phases. The first is prefill which processes the entire input prompt and produces the first output token and the second is decode which generates the rest of output tokens, one-at-a-time. Prefill iterations have high latency but saturate GPU compute due to parallel processing of the input prompt. In contrast, decode iterations have low latency but also low compute utilization because a decode iteration processes only a single token per request. This makes batching highly effective for decodes and consequently for overall throughput. However, batching multiple requests leads to an interleaving of prefill and decode iterations which makes it challenging to achieve both high throughput and low latency. We introduce an efficient LLM inference scheduler, Sarathi-Serve, to address this throughput-latency tradeoff. Sarathi-Serve introduces chunked-prefills which splits a prefill request into near equal sized chunks and creates stall-free schedules that adds new requests in a batch without pausing ongoing decodes. Stall-free scheduling unlocks the opportunity to improve throughput with large batch sizes while minimizing the effect of batching on latency. Furthermore, uniform batches in Sarathi-Serve ameliorate the imbalance between iterations resulting in minimal pipeline bubbles. Our techniques yield significant improvements in inference performance across models and hardware under tail latency constraints. For Mistral-7B on single A100 GPUs, we achieve 2.6x higher serving capacity and up to 3.7x higher serving capacity for the Yi-34B model on two A100 GPUs as compared to vLLM. When used with pipeline parallelism on Falcon-180B, Sarathi-Serve provides up to 5.6x gain in the end-to-end serving capacity. The source code for Sarathi-Serve is available at this https URL https://github.com/microsoft/sarathi-serve . Submission history From: Amey Agrawal view email /show-email/b3656cc7/2403.02310 Mon, 4 Mar 2024 18:47:08 UTC 412 KB \ v1\ /abs/2403.02310v1 Wed, 12 Jun 2024 03:13:20 UTC 1,714 KB \ v2\ /abs/2403.02310v2 v3 Mon, 17 Jun 2024 21:10:46 UTC 1,714 KB References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender IArxiv Recommender What is IArxiv? https://iarxiv.org/about arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .