ReadPaper Blog
WildCity: A Real-World City-Scale Testbed for Rendering, Simulation, and Spatial Intelligence
WildCity introduces a real-world city-scale dataset and testbed for studying whether AI systems can build coherent spatial representations across large urban environments. The paper addresses the shortage of continuous, multimodal, real-world data for city-scale rendering, simulation, and embodied reasoning by processing autonomous-fleet logs into 18 long trajectories covering more than 1,500 km across six cities. Its importance lies in exposing the scalability, extrapolation, and uncertainty challenges that must be solved to build simulation-ready urban digital twins and spatially capable AI agents.
Source: WildCity: A Real-World City-Scale Testbed for Rendering, Simulation, and Spatial Intelligence
WildCity: Can AI Build a City in Its Head?
WildCity is motivated by a central question in spatial AI: whether machine learning systems can internalize the structure of an entire city in a way that supports memory, localization, planning, and reasoning. The paper contrasts this ambition with human spatial cognition, where people gradually form coherent mental maps over tens of square kilometers from partial visual experience. To make this question experimentally grounded, the authors introduce a real-world testbed built from 18 long-horizon trajectories collected across six cities, totaling more than 1,500 km of traversed roads. The dataset is designed for rendering, simulation, and embodied reasoning rather than isolated perception tasks alone. By framing city-scale reconstruction as a first step toward broader spatial intelligence, the paper argues that AI needs long-range, continuous, real-world visual-spatial experience to move beyond small-room or single-block reasoning.
Why Existing Data Falls Short
The paper identifies a data bottleneck as the main reason city-scale spatial intelligence remains difficult to study. Existing embodied AI and vision-language navigation benchmarks often focus on small environments such as rooms, synthetic apartments, or short urban clips, which do not stress long-range spatial coherence. Synthetic datasets provide controllability but introduce a sim-to-real gap that limits their value for photorealistic city simulators and real-world embodied agents. Real-world urban benchmarks, meanwhile, tend to capture isolated scenes or short video segments rather than continuous journeys through diverse city zones. WildCity addresses this gap by emphasizing large-scale, continuous, real-world multimodal observations that preserve the temporal and spatial dependencies needed for studying city-scale memory and reasoning.
What WildCity Provides
WildCity provides continuous surround-view multimodal data collected by autonomous vehicle fleets operating in complex urban environments. Each trajectory is long, with logs averaging about 83.7 km and roughly 2.5 hours of sensory streams, giving models exposure to extended spatial structure rather than fragmented snapshots. The dataset covers distinct functional zones across real cities and includes the messy conditions that reconstruction and perception systems must handle in practice. These in-the-wild challenges include dynamic objects, lighting variation, motion blur, and imperfect camera poses. The paper’s dataset processing pipeline converts raw fleet logs into a form suitable for city-scale reconstruction, simulation, and closed-loop embodied reasoning, making WildCity more than a static collection of images.
From Logs to a Simulator
A major contribution of the paper is an urban-tailored reconstruction baseline that turns WildCity’s fleet logs into simulation-ready city environments. The authors focus on city-scale rendering as the first concrete instantiation of the broader testbed, using 3D Gaussian Splatting as a reconstruction method for large, noisy, unbounded urban scenes. The reconstructed environments are then integrated into a closed-loop simulator, connecting photorealistic rendering with downstream embodied tasks. This pipeline is important because it links raw autonomous-driving-style data to interactive evaluation settings where an agent’s pose, view, and task prompt can affect its future observations. By demonstrating this conversion from real-world logs to reconstructed simulation environments, the paper establishes WildCity as both a dataset and an experimental platform for spatial AI.
The Big Lessons
The paper’s analysis highlights three core obstacles on the path to simulation-ready urban digital twins: scalability, view extrapolation, and data uncertainty. Scalability matters because city-scale trajectories impose computational and memory demands that exceed the assumptions of many scene reconstruction methods. View extrapolation is difficult because agents in a simulator may need to render viewpoints that were not directly observed by the fleet sensors. Data uncertainty arises from real-world imperfections such as moving objects, lighting changes, motion blur, and noisy camera poses, all of which can degrade reconstruction quality and downstream reasoning. The broader implication is that WildCity is intended not merely as a benchmark for prettier urban rendering, but as a catalyst for AI systems that can perceive, remember, and reason across space at a scale closer to human cognition.
