{"slug": "shadowprefill-keeping-your-next-model-warm-for-model-routing-in-agentic-serving", "title": "ShadowPrefill: Keeping Your Next Model Warm for Model Routing in Agentic Serving", "summary": "A November 9, 2026 arXiv paper introduces SHADOWPREFILL, a technique that prefills an agent session's context on the model a per-step router has not yet chosen, using capacity that instances serving normal traffic leave idle, so the next model switch finds its prefix cached. The authors report that model switches drop prefix hit rates from 80-89% on same-model calls to 44-65% and make switch calls 1.5-3.2x slower, and that with LLMRouter routing 24 model pairs over five agentic benchmark suites, SHADOWPREFILL cuts switch-call latency by 8.5-21.4%, LLM-call latency by 3.7-17.7% and routed token cost by 22.5-46.1%, while normal traffic keeps its SLO and energy per request rises 0.4%. The work targets agentic serving workloads, including a production coding-agent trace with a mean context of 63K tokens.", "body_md": "# Computer Science > Distributed, Parallel, and Cluster Computing\n\n  [Submitted on 9 Nov 2026]\n\n# Title:SHADOWPREFILL: Keeping Your Next Model Warm for Per-Step Model Routing in Agentic LLM Serving\n\n[View PDF](https://arxiv.org/pdf/2611.13176)\n\n[HTML (experimental)](https://arxiv.org/html/2611.13176v1)\n\nAbstract:Per-step model routers send each LLM call of an agent to the cost-optimal model dynamically within each session. However, every model switch discards prefix-cache reuse: KV caches are per model, so the new model must prefill the relevant context before it emits a token. Across representative agentic workloads, including a production coding-agent trace with a mean context of 63K tokens, the prefix hit rate falls from 80-89% on same model calls to 44-65% after a switch, and a switch call takes 1.5-3.2x as long. We present SHADOWPREFILL, which prefills the session's context on the model the router has not chosen, in capacity that instances serving normal traffic leave idle, so the next switch finds its prefix cached. SHADOWPREFILL harvests the per-iteration SLO slack of requests in flight, protects routed calls, spills shadow KV to host memory, and falls back to a cold switch whenever the shadow is incomplete. With LLMRouter routing 24 model pairs over five agentic benchmark suites, SHADOWPREFILL cuts switch-call latency by 8.5-21.4%, LLM-call latency by 3.7-17.7%, and routed token cost by 22.5-46.1%, while normal traffic keeps its SLO and energy per request rises by 0.4%.\n    \n\n### Additional Features\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/shadowprefill-keeping-your-next-model-warm-for-model-routing-in-agentic-serving", "canonical_source": "https://arxiv.org/abs/2611.13176", "published_at": "2026-11-13 19:16:25+00:00", "updated_at": "2026-11-13 19:20:14.349305+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-agents", "mlops", "ai-research"], "entities": ["SHADOWPREFILL", "LLMRouter", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/shadowprefill-keeping-your-next-model-warm-for-model-routing-in-agentic-serving", "markdown": "https://wpnews.pro/news/shadowprefill-keeping-your-next-model-warm-for-model-routing-in-agentic-serving.md", "text": "https://wpnews.pro/news/shadowprefill-keeping-your-next-model-warm-for-model-routing-in-agentic-serving.txt", "jsonld": "https://wpnews.pro/news/shadowprefill-keeping-your-next-model-warm-for-model-routing-in-agentic-serving.jsonld"}}