Respond only when the queried action boundary begins or ends.
A fully proactive benchmark for streaming video models and interaction models.
SPOT-Bench requires a streaming model to monitor a live video stream and proactively decide when to respond. Every prediction across the full video is evaluated using the Timeliness-F1 metric.
| # | Model | ABD | PNR | Overall |
|---|---|---|---|---|
| JoyAI-VL-Interaction | 17.0 | 15.6 | 16.3 | |
| MMDuet2 | 14.2 | 5.1 | 9.7 | |
| Qwen2.5-VL (Offline) | 10.0 | 8.1 | 9.1 | |
| Qwen3-VL (Offline) | 10.7 | 7.2 | 9.0 | |
| StreamingVLM | 3.9 | 2.5 | 3.2 |
Streaming video models should respond at the moment an event unfolds, not after the moment has passed. Yet existing online VideoQA benchmarks remain largely retrospective. They pause the video at fixed timestamps, pose questions about current or past events, and score models only at those moments. This protocol leaves streaming outputs untested. To close this gap, we introduce SPOT-Bench, featuring multi-turn proactive queries that evaluate general streaming perception required by an always-on, real-time assistant. SPOT-Bench comes with Timeliness-F1, a consolidated metric that measures streaming outputs by their temporal precision and balanced coverage across the entire video. Our benchmark reveals that during streaming inference: (i) MLLMs detect events reliably in a stream but spam predictions unprompted; (ii) post-training MLLMs for silence reduces spamming but induces unresponsiveness; (iii) half of the streaming video expects no response, which we term dead-time — compute spent here does not affect response latency. These findings motivate Ctrl-SPOT, a training-free streaming controller for MLLMs, that retains their event perception while controlling their streaming outputs for balanced coverage. This establishes a strong baseline on SPOT-Bench.
SPOT-Bench features six proactive streaming tasks grouped into three broad categories.
Detection evaluates a model's general event perception in a streaming setting: can it identify the exact moment an event occurs or transitions, given only partial observations up to that point?
Respond only when the queried action boundary begins or ends.
Respond only when the queried visual state change occurs.
Interaction evaluates a model’s ability to engage in streaming interactions, producing temporally grounded and context-aware responses.
Respond once sufficient visual evidence is available to make a correct decision.
Track procedural progress and proactively cue the next step as the current one nears completion.
Intervention probes proactive behavior, where the model assists a user without explicit requests by anticipating potential errors or failures.
Intervene during a known task or procedure only when the user hesitates or makes an error.
Detect imminent risk and intervene early enough to prevent failure, yet restrained enough to avoid disruptive responses.
@article{chatterjee2026don,
title={Don't Pause! Every prediction matters in a streaming video},
author={Chatterjee, Dibyadip and Pang, Zhanzhong and Sener, Fadime and Song, Yale and Yao, Angela},
journal={arXiv preprint arXiv:2604.24317},
year={2026}
}