What happened
The paper *VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification* introduces a benchmark for testing video multimodal large language models on long-video questions. Its central claim is narrow but important: a model answering a question correctly is not necessarily a model that found the visual evidence needed to support that answer. The authors position the benchmark against evaluations that score answer accuracy alone.
VideoZeroBench uses manually annotated question-answer pairs spanning 13 video domains. According to the paper, the questions are designed to require fine-grained visual cues, fleeting events, and evidence that may be distributed across several video segments. Where applicable, the annotations include temporal intervals and key-frame bounding boxes. This lets an evaluation ask not only whether a system named the right event, object, or action, but also whether it identified when the evidence appeared and where it appeared in a frame.
The authors say every question received two rounds of cross-verification for answer validity and evidence quality. That process is intended to make the benchmark’s evidence labels usable as a separate evaluation target rather than as incidental metadata. The paper does not describe the benchmark as a replacement for all video evaluations; instead, it focuses on a class of failures that answer-only metrics can conceal.
Its five-level diagnostic protocol compares answers produced with and without evidence hints. It then combines answer correctness with independently evaluated temporal and spatial grounding. The design is meant to distinguish a model that can answer from one that can find, localize, and substantiate the basis for its answer. The paper also says it analyzes atomic abilities, evidence spans, input modalities, and “thinking-with-videos” inference to characterize failures beyond a single aggregate number.
Across 19 evaluated models, the paper reports that Gemini-3.7-Flash achieved the highest standard QA accuracy at Level 3, with 24.8%. At Level 5, which jointly requires a correct answer and accurate spatio-temporal localization, no evaluated model exceeded 1.8%. These are the paper’s benchmark results, not general measures of every video model capability or product workflow. The authors state that code and data are publicly released. The arXiv record lists an initial submission on 2 April 2026 and version 3, last revised on 3 November 2026.
What the archive adds
New Runtime’s earlier coverage examined a separate public evaluation of 23 models for correctness and code-review quality, where the leading model cleared every blocking check in 57.7% of attempts (earlier note, 17 October 2026). VideoZeroBench is a new research event rather than a continuation of that evaluation: it applies a comparable concern about verifiable output to evidence in long video.
Trend signal
This supports New Runtime’s verification bandwidth is the scarce engineering resource pattern. A plausible or even correct answer is not enough when a reviewer must inspect the supporting moment in a video. VideoZeroBench makes that distinction measurable by requiring answer correctness and localization together. It does not establish that its score gap will reproduce in every production video workflow.
Why it matters
Teams building video search, investigation tools, monitoring systems, moderation workflows, or video summarization often evaluate a model by whether its final response resembles a reference answer. That can overstate practical reliability. A model may mention the correct event by chance, infer it from surrounding context, or identify an event without being able to direct a user to the relevant moment and visual region.
The paper’s reported gap suggests a more useful test structure for products where evidence needs inspection. Measure answer correctness, temporal localization, and spatial localization separately; then measure their intersection. This is particularly relevant when a product makes claims that a user needs to audit, such as identifying an action in surveillance footage, extracting an event from a meeting recording, or explaining a decision from a long operational video.
The benchmark’s diagnostic framing is also useful. Testing with and without evidence hints can help distinguish retrieval or navigation failures from failures to interpret an already identified clip. The public release of code and data gives teams a starting point for experiments, but it is not proof that adopting the benchmark will predict performance on their own footage, interfaces, or user tasks.
What happens next
- More long-video evaluations will report grounding alongside answer accuracy. (6 months, 60%). Watch for: benchmark leaderboards adding timestamp or region-level scoring.
- Video AI products will expose more inspectable citations to source moments. (12 months, 55%). Watch for: interfaces that attach answers to timestamps and frame references.
Limits and uncertainty
The reported figures cover 19 models on VideoZeroBench, not all available video systems or all real-world video tasks. The arXiv abstract does not provide the complete model roster, per-domain breakdowns, or the detailed settings used for each model, so the top-line scores cannot show which content types account for most failures. The authors describe manual annotation and two rounds of cross-verification, but the paper’s public abstract does not independently establish the error rate of those labels. Nor does the benchmark result demonstrate a causal relationship between its scores and downstream user trust, cost, or reliability in deployed products.
Skill to build now
Add an evidence check to one existing video-AI evaluation this week. For each question, require the system to return an answer, a timestamp range, and—where relevant—a frame region or key frame. Have a reviewer score each field independently before calculating a joint pass rate. This small change can reveal whether a high answer score reflects grounded retrieval or merely a convincing response.
Reader takeaway
For long-video AI, an answer should be treated as a claim, not as proof. VideoZeroBench reports that the best ordinary QA result was 24.8%, while the best result under a joint answer-and-localization requirement was no more than 1.8%. Builders should evaluate whether a system can point to its evidence, not only whether it can state a correct-looking conclusion.
Open question
How well do VideoZeroBench’s joint answer-and-grounding scores predict reliability on production video workloads, especially where users must inspect the cited evidence?