{"slug": "taming-throughput-latency-tradeoff-in-llm-inference-with-sarathi-serve", "title": "Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve", "summary": "Researchers introduced Sarathi-Serve, an LLM inference scheduler that uses chunked-prefills and stall-free scheduling to improve throughput-latency tradeoffs, achieving 2.6x higher serving capacity for Mistral-7B on a single A100 GPU and up to 3.7x for Yi-34B on two A100 GPUs compared to vLLM, with up to 5.6x gain for Falcon-180B with pipeline parallelism. The source code is available on GitHub.", "body_md": "# Computer Science > Machine Learning\n\n  [Submitted on 4 Mar 2024 (\n\n[v1](https://arxiv.org/abs/2403.02310v1)), last revised 17 Jun 2024 (this version, v3)]\n# Title:Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve\n\n[View PDF](/pdf/2403.02310)\n\n[HTML (experimental)](https://arxiv.org/html/2403.02310v3)\n\nAbstract:Each LLM serving request goes through two phases. The first is prefill which processes the entire input prompt and produces the first output token and the second is decode which generates the rest of output tokens, one-at-a-time. Prefill iterations have high latency but saturate GPU compute due to parallel processing of the input prompt. In contrast, decode iterations have low latency but also low compute utilization because a decode iteration processes only a single token per request. This makes batching highly effective for decodes and consequently for overall throughput. However, batching multiple requests leads to an interleaving of prefill and decode iterations which makes it challenging to achieve both high throughput and low latency.\n\nWe introduce an efficient LLM inference scheduler, Sarathi-Serve, to address this throughput-latency tradeoff. Sarathi-Serve introduces chunked-prefills which splits a prefill request into near equal sized chunks and creates stall-free schedules that adds new requests in a batch without pausing ongoing decodes. Stall-free scheduling unlocks the opportunity to improve throughput with large batch sizes while minimizing the effect of batching on latency. Furthermore, uniform batches in Sarathi-Serve ameliorate the imbalance between iterations resulting in minimal pipeline bubbles.\n\nOur techniques yield significant improvements in inference performance across models and hardware under tail latency constraints. For Mistral-7B on single A100 GPUs, we achieve 2.6x higher serving capacity and up to 3.7x higher serving capacity for the Yi-34B model on two A100 GPUs as compared to vLLM. When used with pipeline parallelism on Falcon-180B, Sarathi-Serve provides up to 5.6x gain in the end-to-end serving capacity. The source code for Sarathi-Serve is available at[this https URL](https://github.com/microsoft/sarathi-serve).\n    \n\n## Submission history\n\nFrom: Amey Agrawal [\n[view email](/show-email/b3656cc7/2403.02310)]\n\n**Mon, 4 Mar 2024 18:47:08 UTC (412 KB)**\n\n[\\[v1\\]](/abs/2403.02310v1)\n**Wed, 12 Jun 2024 03:13:20 UTC (1,714 KB)**\n\n[\\[v2\\]](/abs/2403.02310v2)\n**[v3]** Mon, 17 Jun 2024 21:10:46 UTC (1,714 KB)\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\nIArxiv Recommender\n\n*(*[What is IArxiv?](https://iarxiv.org/about))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/taming-throughput-latency-tradeoff-in-llm-inference-with-sarathi-serve", "canonical_source": "https://arxiv.org/abs/2403.02310", "published_at": "2026-09-07 09:00:00+00:00", "updated_at": "2026-09-07 21:30:23.359080+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-infrastructure", "ai-research"], "entities": ["Sarathi-Serve", "Mistral-7B", "Yi-34B", "Falcon-180B", "vLLM", "Microsoft", "A100 GPU"], "alternates": {"html": "https://wpnews.pro/news/taming-throughput-latency-tradeoff-in-llm-inference-with-sarathi-serve", "markdown": "https://wpnews.pro/news/taming-throughput-latency-tradeoff-in-llm-inference-with-sarathi-serve.md", "text": "https://wpnews.pro/news/taming-throughput-latency-tradeoff-in-llm-inference-with-sarathi-serve.txt", "jsonld": "https://wpnews.pro/news/taming-throughput-latency-tradeoff-in-llm-inference-with-sarathi-serve.jsonld"}}