ReadPaper Blog
PanoWorld: Real-World Panoramic Generation
PanoWorld addresses the problem of controllable 360-degree panoramic video generation, where long-range scene memory often breaks under equirectangular distortion, camera rotation, and complex real-world motion. The paper proposes a diffusion-based panoramic world model that decouples rotation from translation, uses Dense Panoramic Ray-Conditioning for motion control, and applies Geometry-aware Memory Augmentation for spatiotemporal consistency. Its importance lies in showing that panoramic generation can become more physically consistent by exploiting the geometry of omnidirectional representations rather than treating panoramas like ordinary perspective videos.
Source: PanoWorld: Real-World Panoramic Generation
Why Panoramas Break Memory
PanoWorld is motivated by a central weakness in panoramic world models: maintaining physical consistency across long camera trajectories in a full 360-degree field of view. The paper argues that existing memory mechanisms, including 3D points and KV-cache-style retrieval, often inherit assumptions from perspective video and therefore misalign past information under equirectangular projection. This misalignment is amplified in panoramic generation because the model must preserve geometry, illumination, and scene layout across the entire surrounding view rather than a narrow crop. The authors frame the problem as both a control challenge and a memory challenge, since a model must follow a target camera trajectory while keeping previously seen content stable. This matters for applications such as autonomous driving, unmanned aerial vehicles, and robotic world modeling, where a generated panoramic sequence must remain plausible over space and time rather than merely produce isolated high-quality frames.
The Gap
The paper’s key geometric insight is that equirectangular panoramas are rotation-equivariant: camera rotation mainly changes the distortion pattern and heading alignment while preserving the underlying scene content. Instead of asking the diffusion model to learn rotation and translation jointly, PanoWorld treats rotation as an implicit geometric transformation and fixes headings so that translation becomes the main learned source of visual change. This decoupling reduces viewpoint chaos in long-horizon generation and directly targets the memory retrieval failures caused by rotation-induced shifts. The authors also connect this design to the properties of ERP imagery, including polar distortion and horizontal continuity, which differ from perspective images and require panoramic adaptation. By simplifying motion in this way, the method turns panoramic geometry from a source of artifacts into a structural prior for controllable generation.
Core Idea
PanoWorld implements this idea as a diffusion-based panoramic world model with two main components: Dense Panoramic Ray-Conditioning and Geometry-aware Memory Augmentation. Dense Panoramic Ray-Conditioning, or DPRC, models current action by using dense panoramic ray information rather than relying on perspective-style pixel motion alone. The paper describes this as aligning camera kinematics with the physical origin of panoramic rays, so that translation-induced parallax can be conditioned in a geometry-aware way. Geometry-aware Memory Augmentation, or GMA, addresses long-range persistence by improving how historical information is retrieved and fused in panoramic space. Together with fixed-heading motion decoupling, these modules are designed to preserve structural and radiometric consistency while still allowing precise trajectory control through complex camera movements.
Data And Training
The paper also introduces World360 because existing panoramic datasets are described as too limited for evaluating real-world physical consistency under large spatial and illumination variation. World360 contains 120,000 high-quality sequences, combining 70,000 curated real-world clips collected with panoramic unmanned aerial vehicles and 50,000 high-fidelity simulated clips from AirSim360. The data pipeline applies rotation decoupling to unify headings, uniform spatial resampling to standardize motion scale by distance rather than time, and illumination filtering to remove sequences with improper exposure. This curation is intended to reduce geometric and photometric noise before model training and evaluation. The authors further describe a three-stage training pipeline that progressively optimizes the components of PanoWorld, including panoramic adaptation, motion modeling, and memory-augmented generation.
Takeaway
The experiments reported in the paper evaluate PanoWorld on World360 and compare it with alternative panoramic or camera-controlled generation approaches such as Matrix3D and OmniRoam. The authors report that PanoWorld outperforms these alternatives by a large margin while achieving high-fidelity visual synthesis, precise trajectory control, and stronger physical consistency in diverse real-world environments. The claimed advantage comes from using panoramic rotation-equivariance and geometry-aware memory rather than relying on expensive explicit reconstruction, frame concatenation, or perspective-oriented retrieval. The paper positions this result as evidence that long-range panoramic world modeling benefits from respecting the structure of omnidirectional representations. Its broader implication is that future controllable video generators for robotics, UAV navigation, and immersive simulation may need geometry-specific memory mechanisms instead of generic video memory modules.
