A Google-led research team has published StreamVibe, a video assistant designed to watch you work and speak up only when help is actually needed. The headline finding is a sober one: on the team's own benchmark, none of the tested models reliably picks the right moment to intervene once videos run longer than about five minutes, because early mistakes compound. The paper, StreamVibe: Proactive Video Agents for Evolving User Intent, went onto arXiv on October 26, 2026, and was first reported in plain language by The Decoder.
TL;DR #
| Question | Answer |
|---|---|
| What is it? | A benchmark (StreamVibe-Bench), a streaming model (StreamVibe-VL) and a memory harness (StreamVibe-Harness) for proactive video agents |
| Who made it? | A Google-led team with academic co-authors, 16 listed authors on the arXiv page | | Benchmark size | 4.1K evaluation samples, 85K training samples | | Headline gains | +10.7% timing and answer quality (model), +12.3% proactive interaction (harness), as reported by the authors | | Biggest weakness | Videos over five minutes: models rarely time responses correctly | | Can I use it? | No code repository is linked in the coverage; see the project page |
What problem does StreamVibe try to solve? #
Almost every video-capable model today is reactive. You ask a question about a clip, it answers. That is fine for summarising a recording but useless for the thing people actually want from a live assistant: a second pair of eyes that notices you are stuck.
Imagine cooking with a camera on. You never say "I forgot to cut off the crust." A good helper would notice you rolling the bread and mention it at that moment, not three steps later and not every ten seconds. The paper calls the underlying difficulty implicit and dynamic intent: what you want is rarely stated, and it changes as the task unfolds.
This is the same direction the industry has been moving in on the product side. We covered Claude Duet as a proactive agent research preview, compared it with OpenAI, Meta and xAI proactive agents, and looked at a hardware proactive assistant launch. StreamVibe is the research-side counterpart: instead of shipping a product, it asks how you would even measure whether such an agent behaves well.
How is StreamVibe built? #
The arXiv abstract describes three pieces.
StreamVibe-Bench
A benchmark covering three kinds of intent: explicit, implicit and dynamic. Per The Decoder, it stages requests in three levels: clear instructions, vague goals, and requests that change over time. The evaluation set is about 4,100 examples drawn from more than 183 hours of everyday-life and software recordings.
The examples were not hand-written from scratch. Gemini 2.5 Flash proposed when help should appear and what it should say, and four human evaluators then had to approve each example unanimously. That is a sensible quality filter, though it also means the benchmark inherits some of the proposing model's view of what good help looks like.
StreamVibe-VL
The model watches a stream and, roughly every second, decides between two actions: stay quiet or speak. The Decoder reports it appears to be based on Alibaba's Qwen3.5 9B. A second training stage uses reinforcement learning to make it more reserved, which directly targets the failure mode every proactive assistant has: interrupting too often.
StreamVibe-Harness
A separate memory component tracks pending requests and completed steps, keeping intents as persistent state in a hierarchical memory. The language model still decides whether to speak and what to say, but it does so with a running record of what the user asked for and what has already been done. The authors report this adds 12.3% on proactive interaction performance.
What are the results? #
| Result | Number | Source |
|---|---|---|
| StreamVibe-VL vs existing baselines (timing and answer quality) | +10.7% | arXiv abstract |
| Harness gain on proactive interaction | +12.3% | arXiv abstract |
| Drop in a key content metric based on the system's own history | up to 61.5% | The Decoder |
| Timing accuracy on videos longer than five minutes | Rarely correct for all models | The Decoder |
Two things stand out. First, the open model beats its base model and other open-source competitors by a wide margin, but according to The Decoder, Google's proprietary Flash model still scores higher in several tests. A small tuned model closing most of a gap is a familiar pattern: we saw it in Coinbase's fine-tuned Qwen3.5 9B fraud agent, where a 9B model beat much larger ones on a narrow task.
Second, the failure on long videos is the real story. When the model conditions on its own earlier outputs, a wrong or badly timed remark makes the next decision worse. A content metric falls by as much as 61.5% in those self-history settings. Short clips look fine; an hour of cooking does not.
Concrete examples from the paper #
The Decoder describes two qualitative cases. In a GIMP tutorial about converting an image to black and white, StreamVibe-VL gives the next step at the right moment, while a competing model (JoyAI) misjudges the state of the task. In a rolled-toast recipe, StreamVibe-VL tells the user to cut off the crust first, the kind of unrequested but welcome nudge the benchmark is built to reward.
These are chosen examples, which is normal for a paper, so treat them as illustrations rather than evidence of average behaviour.
What does this mean if you build agents? #
The practical lesson is about evaluation, not about this particular model.
- Silence is a decision. Most agent evals score the content of an answer. A proactive agent also needs a score forwhether it should have spoken at all . StreamVibe-Bench is one of the first attempts to make that measurable.
- Persistent state beats a long context. The harness result, a 12.3% gain from explicit memory of pending and finished steps, matches what coding-agent builders already see: externalising state outperforms hoping the model remembers. See our guide tohow a Claude Code skill is structured for a related pattern.
- Plan for degradation over time. If your product runs for an hour, test at an hour. Five-minute demos hide the compounding errors this paper documents.
- Small models can be good timers. A 9B model tuned with reinforcement learning to be reserved is cheap enough to run continuously on a stream, which is the only economically realistic way to watch a screen all day.
Questions readers will ask next #
Is this the same as a screen-recording assistant?
Partly. The benchmark mixes real-world recordings (cooking, assembly) with software sessions such as image editing. The model sees a continuous stream, not a single screenshot, and must decide each second whether anything is worth saying. A screen-recording summariser answers after the fact; StreamVibe is judged on whether it spoke at the right instant.
Why does a vague request make the task harder?
With a clear instruction ("tell me when the water boils"), the trigger is checkable. With a vague goal ("help me make this look professional") the agent has to infer sub-goals, notice when one is finished, and avoid repeating advice already followed. That is why the harness keeps a list of pending and completed steps: without it, the model re-suggests things the user has already done.
Should teams trust a 10.7% gain?
Treat it as a direction, not a guarantee. The percentages come from a benchmark the authors built, partly with another Google model proposing the ground truth. That is common practice, but it favours systems that think like the proposer. The right move for a builder is to run a small private test on your own recordings before committing to any proactive design.
What we could not verify #
- The paper's numbers are the authors' own, on their own benchmark. No independent replication exists yet.
- The Decoder's coverage does not link released code or weights. We have not confirmed whether the model, benchmark or harness are downloadable; check the project page and the arXiv listing.
- The arXiv page we read did not list affiliations, so the "Google-led" description rests on The Decoder's report.
- The base-model identity (Qwen3.5 9B) is described by The Decoder as apparent, not stated in the material we read.
How to read it against the product race #
Product teams are racing to ship always-on assistants, and the pitch is usually about what the assistant can do. StreamVibe is a reminder that the hard part is restraint. An assistant that interrupts constantly gets turned off; one that never interrupts is just a chatbot with a camera. The reinforcement-learning stage here, which explicitly trains the model to talk less, is arguably the most transferable idea in the paper.
It also hints at why on-device and open models matter for this category. Watching a stream every second is a continuous workload; sending every frame to a frontier API is expensive and raises obvious privacy questions about who sees your kitchen or your screen. A 9B model that does reasonably well, with a frontier model as an occasional fallback, is a plausible architecture. If you want to follow small-model progress, see our comparison of current small models.
What to watch next #
- A code or weight release from the authors, which would let others reproduce the 10.7% and 12.3% claims.
- Whether other labs adopt StreamVibe-Bench-style intent levels (explicit, implicit, dynamic) in their own proactive-agent evals.
- Long-horizon results: the authors say future work will study how user preferences and the environment influence each other over time, which is exactly the regime where current models fail.
Related reading #
Details reflect the arXiv listing and press coverage as of November 1, 2026 and may change if the authors release new versions.