ReadPaper Blog
GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation
GigaWorld-1 addresses a central bottleneck in embodied robot foundation models: evaluating robot policies is still expensive, hardware-bound, and difficult to scale compared with evaluation in language or vision. The paper proposes a roadmap for using video world models as scalable evaluators, introduces WMBench for measuring evaluator quality, and studies how model choice, action representation, and generated rollouts relate to real-world policy evaluation.
Source: GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation
Research question
GigaWorld-1 is about making robot policy evaluation more scalable by replacing some hardware rollouts with world-model-based evaluation. The paper starts from the observation that embodied robot foundation models are hard to assess because physical trials are costly, slow, and limited by access to real robots and environments. Unlike large language models, whose capabilities can often be benchmarked at scale with static datasets, robot policies must be judged through interactive behavior over time. The proposed direction is to build video world models that can generate policy-conditioned rollouts and serve as evaluators of robot behavior. This matters because scalable evaluation is a prerequisite for faster iteration on robot policies, especially when the policies are meant to operate in diverse real-world settings.
Why old methods fall short
The paper frames existing robot evaluation practice as constrained by the gap between hardware realism and scalable measurement. Real-world rollouts provide the most direct signal about a policy, but they are expensive to collect and difficult to repeat across many policies, tasks, and conditions. Purely offline metrics, by contrast, can miss the temporal and action-dependent structure that determines whether a robot policy actually succeeds. GigaWorld-1 argues that world models tailored to robotics can occupy a useful middle ground if they preserve enough structure, controllability, and action sensitivity to predict meaningful policy outcomes. The work is supported by more than 12,000 hours of training data, indicating that the authors view data scale as part of the roadmap rather than a secondary implementation detail.
Core idea
The core idea in GigaWorld-1 is to evaluate policies through generated rollouts produced by video world models that are conditioned on robot actions or action representations. The paper studies world models not merely as generative video systems, but as potential evaluators whose outputs must remain aligned with the causal structure of robot interaction. This shifts attention from visual plausibility alone to evaluator quality: whether generated trajectories can support conclusions about policy behavior. The authors therefore consider action representation schemes as a central design variable, because the way actions enter the model affects controllability and the relationship between generated outcomes and policy decisions. In this framing, a useful robotics world model must be generative, structured, and sensitive to the policy being evaluated.
Evidence check
To make this evaluation problem concrete, the paper introduces WMBench, a benchmark constructed to assess world models for robotic policy evaluation. WMBench is positioned around evaluator quality, generation quality, structure and controllability, and metrics that connect simulated rollouts to real-world evaluation. The study uses the benchmark to compare multiple video world models and action representations rather than treating a single model architecture as sufficient evidence. This design is important because a world model that produces convincing videos may still fail as an evaluator if it ignores actions, loses task-relevant structure, or gives policy rankings that do not match hardware outcomes. The benchmark therefore serves as both a measurement tool and a way to define what progress should mean for robot-evaluation world models.
One thing to remember
The main empirical contribution is a large-scale analysis of world-model-based policy evaluation using WMBench. The paper reports experiments over 7 video world models, 4 action representation schemes, and more than 324,000 analyzed rollout videos paired with real-world evaluation signals. Those comparisons are used to study how evaluator quality varies across model design choices and how closely generated rollouts can support judgments about robot policies. The named comparison with systems such as GigaWorld-1, Cosmos2.5, and Wan2.2 suggests that the paper treats world-model evaluation as an emerging benchmarkable capability rather than a one-off demonstration. The practical takeaway is that robot policy evaluation can become more scalable, but only if world models are tested against robotics-specific criteria rather than generic video-generation quality alone.
