cd /news/computer-vision/spool-low-latency-streaming-inferenc… · home › topics › computer-vision › article
[ARTICLE · art-171872] src=arxiv.org ↗ pub= topic=computer-vision verified=true sentiment=↑ positive

Spool: Low-Latency Streaming Inference for Hybrid Vision-Language Models

A paper submitted to arXiv on 17 Nov 2026 presents Spool, a training-free framework that caches per-frame state transformations in hybrid vision-language models so expired frames can be removed from recurrent layers without replaying the whole sliding window. Across five open-source hybrid VLMs and four video reasoning benchmarks, Spool posted the lowest p95 time-to-first-token in all 20 model-benchmark tests, cutting it by a geometric mean of 6.6x versus the faster of two recent vLLM baselines while improving accuracy by 2.3 percentage points on average. The work targets latency-sensitive applications such as visual accessibility agents and robotics that must answer questions about recent video over a bounded window of past frames.

recommendation

For sliding-window video inference on hybrid VLMs, cache per-frame affine state transformations and compose them as the window advances instead of re-running expired frames through the backbone.

more in playbook: Inference and self-hosting
read2 min views3 publishedNov 18, 2026
Spool: Low-Latency Streaming Inference for Hybrid Vision-Language Models
Image: source
  [Submitted on 17 Nov 2026]


[View PDF](http://arxiv.org/pdf/2611.23228v1)

[HTML (experimental)](https://arxiv.org/html/2611.23228v1)

Abstract:Applications such as visual accessibility agents and robotics require answering questions about recent video content promptly, over a bounded window of past frames (e.g., the last two minutes). Hybrid vision-language models (VLMs), which interleave attention with linear recurrent layers such as Gated DeltaNet (GDN), are attractive for this setting, but maintaining a sliding window is expensive. The reason is that recurrent layers combine all past frames into a shared state, so expiring old frames normally requires re-running the entire window through the model. We observe that the recurrent update is affine in the state with input-dependent coefficients, so the effect of each frame can be captured as a transformation that composes associatively. We present Spool, a training-free framework that caches per-frame state transformations and composes them efficiently as the window advances, removing expired frames from each recurrent layer's state without replaying frames through the backbone. Removal is exact within each layer; as with key-value eviction in attention, expired frames can retain residual influence through the cached inputs computed by earlier layers. We design a dedicated data structure and GPU kernels to achieve low caching and composition overhead. Across five open-source hybrid VLMs and four video reasoning benchmarks, Spool achieves the lowest p95 time-to-first-token in all 20 model-benchmark tests, reducing it by a geometric mean of $6.6\times$ over the faster of two recent vLLM baselines, while improving accuracy by 2.3 percentage points on average.

Additional Features

References & Citations

...

Bibliographic Explorer

(What is the Explorer?) Connected Papers

(What is Connected Papers?) Litmaps

(What is Litmaps?) scite Smart Citations

(What are Smart Citations?) alphaXiv

(What is alphaXiv?) CatalyzeX Code Finder for Papers

(What is CatalyzeX?) DagsHub

(What is DagsHub?) Gotit.pub

(What is GotitPub?) Hugging Face

(What is Huggingface?) ScienceCast

(What is ScienceCast?) Influence Flower

(What are Influence Flowers?) CORE Recommender

(What is CORE?) arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

── more in #computer-vision 4 stories · sorted by recency
── more on @spool 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/spool-low-latency-st…] indexed:0 read:2min 2026-11-18 · —