ReadPaper Blog
Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory
Light-Omni addresses the latency and cost of agentic video understanding systems that process long, continuous multimodal streams by repeatedly reasoning, searching, and aggregating evidence. The paper proposes a lightweight multimodal memory framework that uses a compact global state and a learned latent state to provide context and retrieval alignment in a single forward pass. Its significance is that long-video agents can become faster and more memory-efficient while preserving or improving accuracy on video understanding benchmarks.
Source: Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory
Research question
The paper studies a central limitation of current Multimodal Large Language Models when they face continuous, long-horizon video: finite context windows make them lose temporal continuity, even when the input contains visual, audio, and textual streams. Light-Omni is motivated by the idea that agentic video understanding needs long-term memory not merely as storage, but as a mechanism for maintaining coherent context across time. The authors frame the task as enabling models to autonomously process and respond to streaming multimodal inputs while retaining past observations and interactions. Their proposed answer is a memory-augmented agent framework that replaces expensive deliberation with reflexive action control. This matters because practical video agents must respond under latency and memory constraints, not only after multi-step searches over fragmented memories.
Why old methods fall short
The paper argues that many existing long-video agents fall short because they depend on “detective-style” iterative reasoning to compensate for weak retrieval. In these systems, video streams are often broken into clips and indexed with coarse metadata such as captions, summaries, or ASR transcripts, which creates a semantic gap between noisy, context-dependent user queries and stored memory representations. Query rewriting, condition generation, and tool-based evidence aggregation can improve results, but they add repeated reasoning steps and external invocations. The authors emphasize that this workflow increases computational cost, latency, and GPU memory use, making responsive agentic interaction difficult. The deeper problem, in their view, is not simply that search needs better prompts, but that memory lacks a persistent global context and retrieval embeddings are poorly aligned with the model’s actual query semantics.
Core idea
Light-Omni’s core idea is to replace repeated reasoning with dual contextual states that make relevant context immediately available. The global state is a finite-sized, non-parametric multimodal script consolidated from episodic memory, using hierarchical merging to preserve recent details while compressing older observations. This state is intended to capture a holistic view of prior visual, auditory, and interactive events, rather than leaving the system to reconstruct context from isolated snippets. Conditioned on that global context, the latent state is a parametric representation that directly drives autonomous actions such as speech or search and produces retrieval embeddings. By jointly learning the backbone and retrieval embedding space, the method aims to bridge the distribution gap between user queries and memory entries without explicit intermediaries such as rewritten queries, generated conditions, or keyword lists.
Evidence check
The method section formalizes Light-Omni as a system that receives an omni-modal input segment at timestamp t, consisting of visual frames, audio signals, and textual instructions, then produces actions while updating structured long-term memory after the response. Its memory system includes user profiles, semantic memory, and episodic components, supporting dynamic knowledge maintenance and efficient retrieval. The framework decouples slow sleep-time consolidation from fast online response: memory can be updated asynchronously, while the active model uses the current dual contextual state for near-immediate action. This design gives general MLLMs such as Qwen2.5-Omni-7B a long-term memory layer without requiring every query to trigger a chain of planning, tool calls, and evidence searches. The important methodological shift is that retrieval becomes a learned, context-conditioned reflex rather than a deliberative reconstruction process.
One thing to remember
The paper reports that Light-Omni improves both accuracy and efficiency across multiple long-video benchmarks, including comparisons with Qwen2.5-Omni-7B and the stronger M3-Agent baseline. In the excerpted results, Light-Omni achieves a 9.5% accuracy gain over Qwen2.5-Omni-7B and a 2.4% gain over M3-Agent, while also delivering 20.5× and 12.1× speedups and 3.3× and 2.6× GPU memory reductions, respectively. These results support the authors’ claim that the cost of prior agentic video systems is not inevitable, but partly a symptom of missing global context and misaligned retrieval. The broader implication is that long-term memory for video agents should be designed as an active contextual substrate, not just a database searched after the fact. Light-Omni therefore positions reflexive, semantically aligned memory access as a route toward scalable real-time multimodal agents.
