{"slug": "a-year-in-llm-serving-workload-evolution-caching-and-load-balancing", "title": "A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing", "summary": "A one-year production trace from Chutes, a cloud platform, reveals how LLM serving workloads evolve over time and how user-model interactions shape traffic, according to a new arXiv paper (2608.13573v1). The study, which captures full production behavior across many models and users, including long-tail models, will release the complete trace to enable downstream research without reliance on sampled or synthetic data.", "body_md": "arXiv:2608.13573v1 Announce Type: new\nAbstract: Large Language Model (LLM) serving has become a critical cloud workload, and realistic traces are essential for motivating and benchmarking serving systems. However, existing LLM serving workload studies remain limited in scale and scope. They often observe short time periods and provide limited visibility into how users interact with models in production. As a result, they do not fully capture how LLM serving workloads evolve over time or how user-model interactions shape production traffic.\nIn this work, we further the understanding of real-world LLM serving workloads through both a global characterization and a longitudinal study of a one-year production trace from Chutes. Unlike prior studies, our trace captures full production behavior across many models and users, including both popular and long-tail models. We analyze the workload from aggregate, temporal, model-level, and user-level perspectives, revealing workload evolution and user-model structure that are typically hidden behind aggregate views. To support future research, we will release the full one-year trace with the paper, enabling downstream studies of production behavior without relying on sampled or synthetically generated workloads.", "url": "https://wpnews.pro/news/a-year-in-llm-serving-workload-evolution-caching-and-load-balancing", "canonical_source": "https://arxiv.org/abs/2608.13573", "published_at": "2026-08-17 04:00:00+00:00", "updated_at": "2026-08-17 04:12:06.752155+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure"], "entities": ["Chutes", "arXiv"], "alternates": {"html": "https://wpnews.pro/news/a-year-in-llm-serving-workload-evolution-caching-and-load-balancing", "markdown": "https://wpnews.pro/news/a-year-in-llm-serving-workload-evolution-caching-and-load-balancing.md", "text": "https://wpnews.pro/news/a-year-in-llm-serving-workload-evolution-caching-and-load-balancing.txt", "jsonld": "https://wpnews.pro/news/a-year-in-llm-serving-workload-evolution-caching-and-load-balancing.jsonld"}}