ReadPaper Blog
Video-Oasis: Rethinking Evaluation of Video Understanding
Video-Oasis argues that current Video-LLM benchmarks can overstate video understanding because many samples can be answered from language priors, audio transcripts, captions, or static visual cues rather than spatio-temporal reasoning. The paper introduces a diagnostic suite that audits existing benchmarks for visual dependency, temporal dependency, and ambiguity, revealing that 55% of sampled benchmark items are shortcut-solvable and that state-of-the-art models perform only marginally above random guessing on the remaining video-native challenges.
Source: Video-Oasis: Rethinking Evaluation of Video Understanding
Why Video Benchmarks Mislead
Video-Oasis addresses a central measurement problem in video understanding: high benchmark scores do not necessarily show that a Video-LLM has understood a video. The paper argues that performance can come from linguistic priors, audio cues, transcript reliance, static context, or single-frame perception rather than grounded spatio-temporal reasoning. This concern matters because modern Video-LLMs are expected to handle fine-grained dynamics and long-form reasoning, yet benchmarks may mix genuinely video-native questions with items solvable without watching temporal evidence. The authors frame video understanding as requiring dependencies that distinguish video from text or images, including temporal continuity, causal interaction, and multi-event narratives. Their motivation is not to add another leaderboard, but to clarify what existing leaderboards are actually measuring.
Video-Oasis Enters
Video-Oasis is proposed as a sustainable diagnostic suite for auditing existing video understanding benchmarks rather than as a new benchmark dataset. Its design centers on three axes: visual-dependency tests, temporal-dependency tests, and ambiguity verification. For visual dependency, the suite removes or abstracts raw visual evidence through Blind, Audio, and Summary probes, testing whether a question can be answered from the prompt alone, from transcribed audio, or from interval-based captions. For temporal dependency, the suite perturbs or removes temporal order to identify samples that do not require reasoning over time. For ambiguity, the paper uses human-in-the-loop inspection to identify annotation problems that arise from the complexity of video data.
The Shortcut Trap
The paper’s large-scale audit spans 14 diverse benchmarks, covering tasks from perception to reasoning and video durations ranging from seconds to hours. Video-Oasis defines a shortcut ratio as the proportion of benchmark samples solvable without visual or temporal dependency. The headline result is that 55% of existing benchmark samples are solvable without visual input or temporal context, indicating that many evaluations contain substantial non-video shortcuts. The paper also reports that benchmarks with higher shortcut prevalence tend to show higher reported accuracy, suggesting that leaderboard gains can be inflated by samples that do not require robust video understanding. This finding reframes benchmark performance as a mixture of genuine spatio-temporal capability and exploitable dataset structure.
When Shortcuts Are Removed
After filtering shortcut-solvable samples, Video-Oasis distills the remaining items into video-native challenges that more directly test temporal continuity, causal interaction, and multi-event narratives. On these distilled challenges, state-of-the-art models cited by the paper perform only marginally above random guessing. This performance drop is central to the paper’s argument because it shows that current Video-LLMs remain weak when superficial pathways are removed. The result also challenges the interpretation of rapid progress on existing benchmarks, especially when gains may come from language reasoning, memorized knowledge, transcript use, or static visual evidence. Video-Oasis therefore positions video-native evaluation as a stricter test of whether models can integrate perception with temporal reasoning.
What the Audit Teaches
The paper’s broader implication is that future video benchmarks should be constructed and audited around shared criteria for visual and temporal grounding. Video-Oasis extends prior benchmark-auditing work such as EgoTempo, Cambrian-S, and Apollo by jointly examining visual dependency, temporal dependency, and ambiguity across a wider evaluation scope. The inclusion of cross-model consensus and manual verification is intended to make shortcut identification more reliable than a single automatic probe. The authors argue that benchmark creators should filter shortcut-solvable cases, verify ambiguity, and preserve video-specific dependencies before treating a dataset as evidence of video understanding. The resulting practical guideline is that evaluation should reward models for grounded spatio-temporal reasoning, not for exploiting linguistic bias, audio transcripts, or isolated frames.
