ReadPaper Blog
Video Generation Models are General-Purpose Vision Learners
The paper argues that large-scale text-to-video generation can serve as a visual counterpart to next-token prediction in NLP: a scalable pre-training objective that teaches spatiotemporal structure, language grounding, and broad visual priors. It introduces GenCeption, a feed-forward perception model built from a pre-trained video generative diffusion backbone and post-trained across multiple tasks, reporting state-of-the-art or competitive results on depth, normals, segmentation, pose, and keypoint prediction with strong data efficiency.
Source: Video Generation Models are General-Purpose Vision Learners
Why Vision Still Feels Fragmented
The paper starts from a central gap in modern computer vision: despite major progress in specialist systems such as Segment Anything for localization and Depth Anything for geometry, the field still lacks a unified general-purpose vision model analogous to large language models. The authors frame this as a search for a visual pre-training objective with the same catalytic role that next-token prediction played in NLP. They argue that such an objective must capture spatiotemporal evolution, align visual representations with language, and scale with data and compute. Static image pre-training is presented as insufficient because the physical world is a 4D continuum involving motion, causality, object permanence, and interactions. This motivation leads the paper to treat generalist vision not as a matter of adding more task-specific heads, but as a pre-training and post-training paradigm problem.
The Proposed Catalyst
The paper’s main proposal is that large-scale text-to-video generation is a strong pre-training paradigm for general-purpose visual intelligence. Text-to-video models must learn to generate coherent video sequences, so the authors argue that their backbones acquire priors about 3D geometry, temporal consistency, physical interactions, and object behavior. Because these models are conditioned on text, they also provide native vision-language alignment rather than requiring a separate semantic grounding stage. The paper further emphasizes scalability: video generation has attracted large datasets and compute because it has relatively low annotation cost and high practical value. GenCeption instantiates this idea by treating a pre-trained video generative diffusion model as the base model for perception rather than as a tool only for synthesis.
How GenCeption Works
GenCeption repurposes the iterative denoising machinery of a video diffusion backbone into a single-step, feed-forward perception model steered by text instructions. During multi-task post-training, the model is fine-tuned primarily with synthetic data across heterogeneous visual tasks, including depth estimation, surface normal prediction, expression-referring segmentation, camera pose estimation, and 3D keypoint prediction. The paper stresses that GenCeption uses a task-agnostic architecture with a unified backbone, unified head, and unified loss style, shifting task specification from architectural redesign to prompts and data formatting. This design contrasts with systems that require task-specific encoders, decoders, or losses for each visual problem. The resulting framework is intended to make adding new perception tasks more like expanding an instruction-following dataset than building a new model family.
What the Paper Reports
The empirical section reports that GenCeption achieves state-of-the-art or broadly competitive performance across a diverse suite of perception tasks while using a single generalist model. The paper compares it with specialized models including DepthAnything V3, SAM3, D4RT, VGGT-Ω, Sapiens, David, Genmo, and Lotus-2, and states that GenCeption often matches or surpasses these task-focused systems. It also evaluates pre-training choices and reports that the video generative diffusion backbone outperforms alternative video representation learning paradigms such as V-JEPA and VideoMAE V2 under comparable fine-tuning settings. On depth estimation, the authors describe preliminary scaling behavior, with performance improving as data and model size increase. They also report substantial data efficiency, with GenCeption reaching comparable performance to leading models such as D4RT and VGGT-Ω using 7× to 500× less training data.
The Weird Ending
The paper’s broader implication is that video generation may be a foundation for perception rather than merely a synthesis technology. One of its most striking claims is that GenCeption exhibits emergent behavior: a model trained exclusively on synthetic human videos can generalize to real-world footage and to out-of-distribution object categories such as animals and robots. The authors interpret this as evidence that generative video pre-training can encode transferable physical-world priors, not just appearance statistics from the training distribution. This matters because sim-to-real transfer and category generalization are central obstacles for vision systems intended to operate in open environments. The conclusion positions text-to-video generative pre-training as a plausible path toward unified, instruction-steered, generalist vision intelligence for the physical world.
