ReadPaper Blog
LongE2V: Long-Horizon Event-based Video Reconstruction, Prediction, and Frame Interpolation with Video Diffusion Models
LongE2V addresses the problem of turning sparse, asynchronous event-camera streams into coherent, human-interpretable video across reconstruction, long-horizon prediction, and frame interpolation. The paper fine-tunes a pretrained video diffusion model, CogVideoX, and conditions it on event voxel grids so that video priors supply photorealistic structure while events provide precise motion guidance. Its significance is that a single architecture improves perceptual quality, temporal coherence, robustness across sensor resolutions, and zero-shot interpolation behavior compared with prior task-specific methods.
Why Event Video Is So Annoying
LongE2V starts from a central difficulty in event-based vision: event cameras record asynchronous brightness changes rather than dense intensity images, so recovering full video is an ill-posed inverse problem. The paper argues that this difficulty appears in three related tasks: reconstructing high-fidelity video from event streams, predicting long sequences from a start frame plus events, and interpolating intermediate frames from start and end frames plus event dynamics. Earlier CNN, RNN, Transformer, and hypernetwork methods such as E2VID and FireNet aggregate event information, but regression-style training often produces blurred textures because it averages plausible visual outcomes. The authors also identify instability in naive diffusion-based long sequence generation and ghosting in difficult event-based frame interpolation. By framing all three problems as conditional event-based video generation, the paper positions LongE2V as a bridge between neuromorphic sensing and conventional video representations.
The Core Trick
The core method is to reuse the visual prior of a pretrained video diffusion model rather than learning the entire video formation problem from scratch. LongE2V builds on CogVideoX I2V, whose 3D VAE compresses video latents and whose Diffusion Transformer learns a denoising objective over noisy latent videos conditioned on external signals. The paper converts raw events of the form x, y, timestamp, and polarity into event voxel grids by accumulating signed brightness-change events across temporal bins with linear interpolation. These event voxels become conditioning signals that guide the diffusion model toward motion consistent with the sensor stream while the pretrained video prior contributes photometric detail and natural video statistics. This design is presented as a source of data efficiency and perceptual quality, because the model does not have to relearn the appearance of videos solely from event supervision.
Keeping Long Sequences From Falling Apart
For long-horizon prediction, the paper focuses on error accumulation, a common failure mode when generated frames are repeatedly fed forward over time. It reports that direct video diffusion can suffer severe temporal drift in long sequences, including color instability, because small deviations compound across autoregressive steps. LongE2V introduces Autoregressive Unrolling so training better matches the inference setting in which predictions must condition later predictions. It combines this with Adaptive Context Switching, which dynamically updates temporal dependencies rather than relying on a fixed context schedule. The intended effect is to preserve temporal coherence over extremely long event-guided sequences while still using the generative capacity of the diffusion model.
Interpolation Without Ghosts
For event-based video frame interpolation, the paper addresses the mismatch between precise event timing and the latent temporal structure imposed by the video model's 3D VAE. Existing interpolation methods can struggle under fast or complex motion, producing ghosting artifacts when intermediate content is not aligned with bidirectional evidence from the input frames and events. LongE2V proposes Reencoding Alignment to reduce temporal misalignment in the latent space, making the synthesized intermediate trajectory more consistent with the encoded video representation. It further uses Cross Residual Correction to enforce more precise bidirectional consistency between the available endpoints and the generated frames. This mechanism extends the pretrained diffusion prior beyond ordinary video synthesis into zero-shot event-guided interpolation.
Why It Generalizes
The paper also treats robustness across event sensors as a practical requirement rather than an afterthought. Because event voxel density can change with sensor resolution and scene dynamics, LongE2V introduces Event Voxel Density Augmentation to expose the model to varying event densities during training. This augmentation is designed to reduce sensitivity to a particular event-camera configuration and improve generalization when deployment data differs from the training distribution. In experiments on real-world benchmarks, the authors report that LongE2V outperforms state-of-the-art baselines across reconstruction, prediction, and interpolation, with stronger perceptual quality, temporal coherence, and zero-shot generalization. The broader implication is that pretrained video diffusion priors can serve as flexible foundations for event-based computational photography when paired with task-specific temporal conditioning and stability mechanisms.
