---
type: "post"
id: "videozerobench-tests-whether-video-multimodal-models-can-identify-280cbc"
slug: "videozerobench-tests-whether-video-multimodal-models-can-identify-280cbc"
title: "VideoZeroBench tests whether video multimodal models can identify the frames and pixels supporting an answer."
description: "VideoZeroBench tests long-video models on both answering questions and locating the supporting evidence."
retrieval_nugget: "VideoZeroBench tests long-video models on both answering questions and locating the supporting evidence. Its authors report 24.8% as the best standard QA accuracy across 19 evaluated models, but no more than 1.8% when correct answers must also be grounded in the right time and image regions. For builders, the gap puts evidence localization—not just fluent answers—under scrutiny."
published_at: "2026-11-07"
updated_at: "2026-11-07"
record_date: "2026-11-07"
date_kind: "published_at"
topics: ["models"]
entities: ["VideoZeroBench","Gemini-3.7-Flash"]
related_patterns: []
novelty_label: "new_fact"
what_changed: "VideoZeroBench adds a long-video benchmark showing that models’ accuracy falls to 1.8% when answers must be supported by correct spatio-temporal localization."
prior_refs: []
source_url: "https://marinero4972.github.io/projects/VideoZeroBench"
source_urls: ["https://marinero4972.github.io/projects/VideoZeroBench"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:videozerobench-tests-whether-video-multimodal-models-can-identify-280cbc"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/videozerobench-tests-whether-video-multimodal-models-can-identify-280cbc/","markdown":"https://newruntime.com/posts/videozerobench-tests-whether-video-multimodal-models-can-identify-280cbc.md","json":"https://newruntime.com/posts/videozerobench-tests-whether-video-multimodal-models-can-identify-280cbc.json"}
markdown_url: "https://newruntime.com/posts/videozerobench-tests-whether-video-multimodal-models-can-identify-280cbc.md"
---

# VideoZeroBench tests whether video multimodal models can identify the frames and pixels supporting an answer.

## Retrieval answer

VideoZeroBench tests long-video models on both answering questions and locating the supporting evidence. Its authors report 24.8% as the best standard QA accuracy across 19 evaluated models, but no more than 1.8% when correct answers must also be grounded in the right time and image regions. For builders, the gap puts evidence localization—not just fluent answers—under scrutiny.

## Summary
VideoZeroBench tests long-video models on both answering questions and locating the supporting evidence. Its authors report 24.8% as the best standard QA accuracy across 19 evaluated models, but no more than 1.8% when correct answers must also be grounded in the right time and image regions. For builders, the gap puts evidence localization—not just fluent answers—under scrutiny.

## What happened
The VideoZeroBench team introduces a long-video understanding benchmark built around manually annotated question-and-answer pairs. The benchmark spans 13 video domains and targets fine-grained cues, fleeting events, and evidence distributed across multiple segments. Questions may require models to find information that is sparse in a long clip or separated across time. The authors say temporal intervals and key-frame bounding boxes are annotated where applicable, and that questions undergo two rounds of cross-verification for answer validity and evidence quality.

The benchmark contains 500 questions, and the reported model accuracies use the same 500-question set. It evaluates 19 models using a five-level diagnostic protocol. The levels distinguish answering from increasingly demanding evidence checks: the protocol compares performance with and without evidence hints, then combines correctness with independently evaluated temporal and spatial grounding. In the authors’ account, this is meant to expose whether a model can do more than produce a correct-sounding answer: it must also locate the time span and visual region that support it.

The strongest standard question-answering result reported is 24.8% at Level 3, achieved by Gemini-3.7-Flash. At Level 5, where a response must be correct and its spatio-temporal localization accurate, no model exceeds 1.8%. The page says most open-source video multimodal large language models score zero at Level 5. These are benchmark results, not a claim that models never answer long-video questions correctly; they show how results change under the benchmark’s grounding requirement.

The five levels are designed as a diagnostic rather than a single leaderboard score. VideoZeroBench says it first tests answering with and without evidence hints, then evaluates temporal and visual grounding independently before applying the joint Level-5 requirement. Its leaderboard also reports temporal intersection-over-union over 442 temporally annotated questions and visual intersection-over-union over 372 spatially annotated questions. Human scores, the page says, use a randomly sampled 50-question subset. Those details make the evaluation narrower and more inspectable than a simple answer-only comparison.

The authors also analyze atomic abilities, evidence spans, input modalities, and “thinking-with-videos” inference. Their summary says agentic thinking with video helps, but remains limited by grounding precision. The benchmark covers 11 atomic capabilities grouped into detailed perception, spatial and temporal reasoning, and semantic and cross-modal reasoning. The researchers say the code and data are publicly released. The project page links to the paper, dataset, code, leaderboard, and examples; it does not state a price or access restrictions.

## What the archive adds
New Runtime's earlier coverage of [Artificial Analysis's Code2Video-AA benchmark](https://newruntime.com/posts/artificial-analysis-launched-code2video-aa-benchmarking-27-models-0bbef6/) described a comparison of 27 models on 114 HTML-rendered marketing-video briefs. That benchmark tests a different task: generating rendered marketing videos from briefs rather than answering questions about long videos. VideoZeroBench is a new event and adds a separate evaluation lens: whether answers can be tied to the relevant temporal and spatial evidence.

## Trend signal
This does not yet fit a pattern New Runtime tracks. VideoZeroBench is a research benchmark about video-question answering and grounding; the closest research pattern concerns scientific-agent benchmarks that grade inspectable artifacts, which is not the same problem. The result may inform future discussion of evidence-grounded model evaluation, but this benchmark alone does not establish a broader trend.

## Why it matters
For a team building video search, review, or monitoring, an answer without a verifiable time span can be difficult to audit or act on. If a system says an event occurred but cannot point to the segment—and, where relevant, the pixels that support the claim—the user still has to search the footage manually. That can erase much of the value of automated analysis, particularly when clips are long and evidence is fleeting.

The benchmark’s gap also changes how teams should read model scores. A system that performs better on ordinary question answering is not automatically better at producing evidence a reviewer can check. Evaluation should separate answer quality from evidence retrieval and localization, rather than treating one as a proxy for the other. VideoZeroBench offers a concrete test set and a five-level structure for examining that distinction.

This matters most where the answer becomes a decision input: flagging an event in a recording, retrieving a moment for a reviewer, or explaining why an alert was raised. A product can present the answer together with a clip interval and a visual marker, then make those artifacts part of review rather than relying on a textual assertion alone. The reported results are a reason to measure that workflow directly before depending on grounded video answers in production.

## Limits and uncertainty
These figures describe performance on VideoZeroBench’s 500 questions, not all video understanding or every deployment setting. The project page reports aggregate accuracies but does not, in its headline findings, provide confidence intervals or establish how the ranking transfers to a team’s own video sources and tasks. The 1.8% ceiling applies to the Level-5 criterion combining answer correctness and spatio-temporal localization; it should not be read as ordinary QA accuracy.

The authors describe manual annotation, cross-verification, and independent grounding evaluation, but the benchmark results have not been independently reproduced here. Aggregate figures also leave open which video domains, evidence-span lengths, or atomic abilities drive the largest failures for a particular model. The page identifies Gemini-3.7-Flash as the Level-3 leader, but does not establish whether later model versions, different sampling budgets, or a production retrieval pipeline would change the same ranking.

## Reader takeaway
VideoZeroBench’s central finding is the gap: models can answer some long-video questions, but almost none meet the benchmark’s joint requirement to answer correctly and point to the supporting time and image regions. If your product depends on trustworthy video analysis, test evidence localization as its own capability—not just answer accuracy.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Models](https://newruntime.com/topics/models.md): Explore the models topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)