{"slug": "tidecache-efficient-hybrid-llm-serving-with-transparent-cache-management-and", "title": "TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming", "summary": "A November 19, 2026 arXiv paper presents TideCache, a serving system for hybrid open-weight language models that completes up to 2.04x as many agent turns as SGLang with HiCache. Implemented in SGLang and evaluated on Nvidia GB300 and GH200 hardware across seven model/platform configurations, TideCache cuts overall P50 per-token latency by 20–59% and improves agent P50 time-to-first-token by 1.40–3.05x. The system combines a Transparent Cache Pool that redistributes physical pages behind fixed virtual addresses, a Cache Streamer that prefetches the next global-attention layer's prefix into shared GPU slots during bounded-state layer computation, and a Request Scheduler that picks resident or streamed execution based on request size and queue state.", "body_md": "# Computer Science > Distributed, Parallel, and Cluster Computing\n\n  [Submitted on 19 Nov 2026]\n\n# Title:TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming\n\n[View PDF](http://arxiv.org/pdf/2611.26619v1)\n\n[HTML (experimental)](https://arxiv.org/html/2611.26619v1)\n\nAbstract:Hybrid architectures have become mainstream among open-weight language models. Their global-attention and bounded-state layers maintain heterogeneous caches whose demands vary with context length and concurrency. We present TideCache, a serving system on CPU--GPU superchips with three components for allocating GPU memory among caches and managing their residency across GPU and host memory. The Transparent Cache Pool redistributes physical pages among cache components behind fixed virtual addresses, preserving cache layouts and backend interfaces. The Cache Streamer uses computation intervals provided by bounded-state layers to prefetch the next global-attention layer's prefix into GPU slots shared across global-attention layers, reducing GPU residency while overlapping layer-wise transfers with computation. Generated KV remains resident. The Request Scheduler selects resident or streamed execution and triggers capacity adjustments according to request size and queue state. Together, they let HBM released by streaming accommodate other cache components and additional requests. Implemented in SGLang and evaluated on GB300 and GH200 across seven model/platform configurations, TideCache completes up to 2.04x as many agent turns, reduces overall P50 per-token latency by 20--59%, and improves agent P50 TTFT by 1.40--3.05x compared with SGLang with HiCache.\n    \n\n### Additional Features\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/tidecache-efficient-hybrid-llm-serving-with-transparent-cache-management-and", "canonical_source": "http://arxiv.org/abs/2611.26619v1", "published_at": "2026-11-20 14:37:43+00:00", "updated_at": "2026-11-20 15:13:07.278557+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-research", "mlops", "ai-agents"], "entities": ["TideCache", "SGLang", "HiCache", "Nvidia GB300", "Nvidia GH200", "arXiv"], "also_reported_by": [], "takeaway": "TideCache streams global-attention prefixes into GPU slots shared across layers during bounded-state layer computation, completing up to 2.04x as many agent turns and cutting P50 per-token latency 20–59% versus SGLang with HiCache.", "recommendation": {"advice": "Prefetch the next global-attention layer's prefix into shared GPU slots during bounded-state layer computation to free HBM for other caches; the paper reports up to 2.04x more agent turns and 20–59% lower P50 per-token latency.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/tidecache-efficient-hybrid-llm-serving-with-transparent-cache-management-and", "markdown": "https://wpnews.pro/news/tidecache-efficient-hybrid-llm-serving-with-transparent-cache-management-and.md", "text": "https://wpnews.pro/news/tidecache-efficient-hybrid-llm-serving-with-transparent-cache-management-and.txt", "jsonld": "https://wpnews.pro/news/tidecache-efficient-hybrid-llm-serving-with-transparent-cache-management-and.jsonld"}}