ReadPaper Blog
Parallelized Autoregressive Decoding for Omni-Modal Dense Video Captioning
The paper addresses dense video captioning, where a model must localize multiple events in an untrimmed video and generate temporally grounded descriptions for each event. It proposes PadCaptioner, a parallelized autoregressive framework that uses latent global event planning and event-factorized decoding to reduce token-by-token latency while preserving caption quality and temporal grounding. The result matters because modern video large language models are powerful but slow for long, event-dense videos, and the paper reports both stronger grounding and captioning performance and a 3.8× wall-time decoding speedup.
Source: Parallelized Autoregressive Decoding for Omni-Modal Dense Video Captioning
Why dense video captioning is hard
Dense video captioning is difficult because it combines two demanding tasks: identifying multiple meaningful temporal events in an untrimmed video and producing a caption for each localized event. The paper emphasizes that this differs from conventional video captioning, which often generates a single global description, because DVC must produce many temporally grounded outputs for long-form content. Autoregressive video large language models have become a strong paradigm for this setting because they unify video and language modeling and benefit from pretrained multimodal reasoning. Their weakness, however, is that they decode one token at a time, so long videos with many events can require hundreds of sequential generation steps. The paper frames this latency as a scalability bottleneck for applications such as egocentric perception, video narration, embodied agents, search and indexing, and multi-event video generation.
The missing idea
The central observation behind the paper is that strict left-to-right dependency is not always necessary across distinct video events. Caption tokens within the same event often need sequential generation to preserve local semantic coherence, but tokens associated with temporally separate events may have weak local dependencies on one another. PadCaptioner exploits this structure by restructuring the causal dependency graph rather than abandoning autoregressive modeling altogether. This lets the framework keep the strengths of autoregressive decoding where they matter most while allowing independent event-level caption streams to progress in parallel. The paper positions this approach as a way to avoid the quality-efficiency tradeoff that can arise in diffusion-based language models and other non-autoregressive strategies.
PadCaptioner’s plan
PadCaptioner first performs latent global planning, in which the model generates compact global event tokens denoted as G1 through GK. Each global event token is intended to represent one temporally coherent event and to encode the inter-event causal structure needed before detailed caption generation begins. This planning stage is autoregressive, so the model can preserve temporal coordination among events rather than treating them as unrelated fragments. The event tokens also adaptively aggregate audio-visual semantics from the corresponding regions of the video, using both visual and synchronized audio information. By converting raw video and audio streams into a planned sequence of event-level representations, the method creates the scaffold needed for later parallel decoding.
Parallel, but still grounded
After global planning, the paper introduces event-factorized parallel decoding to generate captions more efficiently while remaining grounded. The dependency graph is decomposed into multiple event-conditioned subchains, each anchored by a corresponding global event token. Subchains can be decoded in parallel across events, but generation remains autoregressive within each event so that local word order and semantic consistency are preserved. The method also adjusts inter-token visibility so local tokens from different event subchains do not attend to one another unnecessarily, reducing cross-event interference. At the same time, all subchains retain access to the shared global event tokens, allowing each caption to remain aware of the broader event structure.
What the paper reports
The experiments reported in the paper evaluate PadCaptioner on dense video captioning and other temporally grounded omni-modal video understanding tasks. The excerpt highlights gains on LongVALE, where the method surpasses the previous state of the art in both event grounding and captioning while improving inference efficiency. The paper reports a 3.8× actual wall-time decoding speedup, along with additional efficiency gains measured per token, showing that the dependency restructuring has practical runtime impact rather than only theoretical parallelism. It also reports improved event grounding F1 and captioning similarity, suggesting that latent planning and event-factorized decoding help accuracy as well as speed. The broader implication is that autoregressive multimodal LLMs can be made more scalable for long, dense videos without discarding their pretrained language and cross-modal reasoning capabilities.
