ReadPaper Blog
Enhancing In-context Panoramic Generation via Geometric-aware Pretraining
The paper introduces Canvas360, a two-stage framework for in-context panoramic generation that tackles geometric inconsistency in equirectangular 360-degree images. It combines geometry-aware RGB-depth pretraining, velocity circular padding, and unified task fine-tuning on Canvas360Dataset, a 1M-sample dataset for text-to-panorama generation, style transfer, inpainting, outpainting, and editing. The reported experiments show stronger panorama-specific fidelity, better boundary consistency, and especially strong performance on the FAED metric.
Source: Enhancing In-context Panoramic Generation via Geometric-aware Pretraining
The Panorama Problem
The paper addresses a central failure mode in panoramic image generation: equirectangular projection represents a spherical scene on a rectangle, which creates latitude-dependent distortions that can break geometry during editing. This matters because in-context panoramic generation must condition on both user-provided images and text prompts while preserving a coherent 3D scene structure across the full 360-degree field of view. Prior panoramic methods have used multi-view stitching, cube-map representations, spherical-aware convolutions, or 3D spherical positional embeddings, but the authors argue that these approaches still struggle with boundary artifacts and geometric consistency. Canvas360 is proposed as a way to learn a stronger panoramic prior before adapting to downstream in-context tasks. The paper frames the core problem as not merely generating plausible pixels, but maintaining spherical continuity, object distortion details, and global scene coherence in ERP panoramas.
The Missing Piece
A key motivation of the paper is that depth priors from ordinary perspective image generation do not transfer cleanly to panoramas. In perspective images, depth is typically defined along the Cartesian Z-axis, while panoramic scenes are naturally spherical and depth corresponds to radial distance from the camera center. This difference makes geometry-aware training for ERP panoramas a distinct problem rather than a direct reuse of planar image methods. The authors note that existing distortion-aware editing designs, including cube-map-based editing and spherical positional encoding approaches, can still fail to preserve the underlying 3D structure. Canvas360 therefore asks how depth can be formulated and learned under spherical geometry so that panoramic generation remains consistent at image boundaries and across distorted regions.
Stage 1: Teach Geometry
The first stage of Canvas360 is geometry-aware pretraining on 100K annotated RGB-depth panoramas. The method pairs RGB panoramas with depth predictions, encodes both modalities into latent tokens, concatenates the latents, and trains a Flow Transformer with flow-matching objectives for both RGB and depth generation. Positional offsets and a similarity loss are used to regulate the relationship between RGB and depth representations so the model can exploit geometric correspondence without collapsing the modalities into an indistinct representation. The paper also introduces velocity circular padding, which enforces spherical continuity by handling the left-right boundary of ERP panoramas as a connected region rather than a hard image edge. This pretraining stage is intended to give the model a transferable panoramic prior before downstream task-specific supervision is applied.
Stage 2: One Model, Four Jobs
The second stage fine-tunes Canvas360 as a unified in-context panoramic generation model after discarding depth inputs. The paper introduces Canvas360Dataset, a 1M-scale dataset consisting of 100K panoramic RGB-depth samples and 900K paired downstream samples for four in-context tasks: 250K outpainting, 250K inpainting, 200K style transfer, and 200K panorama editing. The seed data include 100K indoor and outdoor scenes built from existing resources and state-of-the-art generation models, then expanded through a scalable data synthesis pipeline. For downstream conditioning, the model uses token-level concatenation to combine heterogeneous contextual inputs in a single framework. This design lets one model support multiple panoramic generation tasks rather than requiring separate architectures or task-specific pipelines.
What It Achieved
The experiments evaluate Canvas360 on text-to-panorama generation, style transfer, inpainting, outpainting, and editing. The paper reports that Canvas360 improves panoramic image fidelity and boundary consistency, with particularly strong results on the panorama-specific FAED metric and competitive or leading scores across the quantitative evaluations described in the work. The qualitative examples are used to support the claim that the model captures a rich panoramic prior and transfers it to diverse in-context applications. The broader implication is that large-scale geometry-aware pretraining can be more effective than relying only on downstream in-context task training for ERP panoramas. The paper also positions Canvas360Dataset as an important infrastructure contribution for scaling panoramic generation research beyond data-limited, task-fragmented systems.
