{"slug": "spool-low-latency-streaming-inference-for-hybrid-vision-language-models", "title": "Spool: Low-Latency Streaming Inference for Hybrid Vision-Language Models", "summary": "A paper submitted to arXiv on 17 Nov 2026 presents Spool, a training-free framework that caches per-frame state transformations in hybrid vision-language models so expired frames can be removed from recurrent layers without replaying the whole sliding window. Across five open-source hybrid VLMs and four video reasoning benchmarks, Spool posted the lowest p95 time-to-first-token in all 20 model-benchmark tests, cutting it by a geometric mean of 6.6x versus the faster of two recent vLLM baselines while improving accuracy by 2.3 percentage points on average. The work targets latency-sensitive applications such as visual accessibility agents and robotics that must answer questions about recent video over a bounded window of past frames.", "body_md": "# Computer Science > Computer Vision and Pattern Recognition\n\n  [Submitted on 17 Nov 2026]\n\n# Title:Spool: Low-Latency Streaming Inference for Hybrid Vision-Language Models\n\n[View PDF](http://arxiv.org/pdf/2611.23228v1)\n\n[HTML (experimental)](https://arxiv.org/html/2611.23228v1)\n\nAbstract:Applications such as visual accessibility agents and robotics require answering questions about recent video content promptly, over a bounded window of past frames (e.g., the last two minutes). Hybrid vision-language models (VLMs), which interleave attention with linear recurrent layers such as Gated DeltaNet (GDN), are attractive for this setting, but maintaining a sliding window is expensive. The reason is that recurrent layers combine all past frames into a shared state, so expiring old frames normally requires re-running the entire window through the model. We observe that the recurrent update is affine in the state with input-dependent coefficients, so the effect of each frame can be captured as a transformation that composes associatively. We present Spool, a training-free framework that caches per-frame state transformations and composes them efficiently as the window advances, removing expired frames from each recurrent layer's state without replaying frames through the backbone. Removal is exact within each layer; as with key-value eviction in attention, expired frames can retain residual influence through the cached inputs computed by earlier layers. We design a dedicated data structure and GPU kernels to achieve low caching and composition overhead. Across five open-source hybrid VLMs and four video reasoning benchmarks, Spool achieves the lowest p95 time-to-first-token in all 20 model-benchmark tests, reducing it by a geometric mean of $6.6\\times$ over the faster of two recent vLLM baselines, while improving accuracy by 2.3 percentage points on average.\n    \n\n### Additional Features\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/spool-low-latency-streaming-inference-for-hybrid-vision-language-models", "canonical_source": "http://arxiv.org/abs/2611.23228v1", "published_at": "2026-11-18 15:29:17+00:00", "updated_at": "2026-11-18 15:43:20.979563+00:00", "lang": "en", "topics": ["computer-vision", "machine-learning", "ai-research", "large-language-models", "ai-infrastructure"], "entities": ["Spool", "Gated DeltaNet", "vLLM", "arXiv"], "also_reported_by": [], "takeaway": "Spool's per-frame affine state caching removes expired frames from hybrid VLM recurrent layers without replaying the window, cutting p95 time-to-first-token 6.6x geometrically versus the faster of two vLLM baselines and gaining 2.3 accuracy points across 20 tests.", "recommendation": {"advice": "For sliding-window video inference on hybrid VLMs, cache per-frame affine state transformations and compose them as the window advances instead of re-running expired frames through the backbone.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/spool-low-latency-streaming-inference-for-hybrid-vision-language-models", "markdown": "https://wpnews.pro/news/spool-low-latency-streaming-inference-for-hybrid-vision-language-models.md", "text": "https://wpnews.pro/news/spool-low-latency-streaming-inference-for-hybrid-vision-language-models.txt", "jsonld": "https://wpnews.pro/news/spool-low-latency-streaming-inference-for-hybrid-vision-language-models.jsonld"}}