ReadPaper Blog
RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies
RoboDojo addresses the lack of comprehensive, reproducible evaluation for generalist robot manipulation policies by combining scalable simulation testing with standardized real-world deployment assessment. The benchmark introduces 42 simulation tasks, 18 real-world tasks, RoboDojo-RealEval for remote physical evaluation, XPolicyLab for unified policy integration, and a public leaderboard evaluating 30 policies, making it easier to diagnose where current robot policies succeed and fail.
Research question
RoboDojo is about measuring whether generalist robot manipulation policies can actually handle the diverse demands of physical manipulation, rather than only succeeding on narrow or conveniently structured tasks. The paper argues that progress in embodied foundation models and language-conditioned manipulation has made evaluation a bottleneck: researchers need to know not just whether a policy can complete a task, but which capabilities it has learned and which failure modes remain. RoboDojo frames this as a unified sim-and-real benchmarking problem, because simulation offers fast, scalable feedback while real-world testing exposes contact-rich dynamics, perception noise, actuation errors, and environmental variation. Its central contribution is a benchmark with 42 simulation tasks and 18 real-world tasks designed to cover complementary manipulation challenges. The result matters because it turns evaluation into a diagnostic loop for generalist policies, connecting model iteration in simulation with reproducible evidence from physical deployment.
Why old methods fall short
The paper identifies two major weaknesses in existing robot manipulation benchmarks: limited task diversity and a persistent split between simulation-only and real-world-only evaluation. Many prior benchmarks test short-horizon or skill-narrow behaviors, and even when they vary objects, layouts, or language, they often preserve similar underlying manipulation patterns. RoboDojo instead emphasizes capability dimensions such as generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following, which are meant to expose qualitatively different challenges. The authors also stress that simulation cannot fully capture deployment realities, while real-world evaluation is expensive and hard to reproduce without standardized hardware, layouts, lighting, reset procedures, and interfaces. This gap motivates RoboDojo’s design as a benchmark that uses simulation for broad, efficient diagnosis and real-world evaluation for controlled validation under physical conditions.
Core idea
The core mechanism of RoboDojo is a shared evaluation ecosystem rather than a single task suite. Its simulation benchmark runs in Isaac Sim and is organized around five capability dimensions: Generalization, Memory, Long-Horizon, Precision, and Open. Its real-world benchmark complements those diagnostics with 18 tasks across three robot embodiments, exposing policies to deployment conditions that simulation alone may miss. RoboDojo-RealEval standardizes the physical setup, scene reset, evaluation protocol, and deployment interface, while remote cloud access is intended to make real-world testing more reproducible and accessible. XPolicyLab provides the integration layer, allowing policies to be connected once and evaluated across both simulated and real-world settings with minimal policy-side adaptation.
Evidence check
The paper’s evidence comes from integrating 30 robot manipulation policies into XPolicyLab and evaluating them through RoboDojo’s simulation and real-world benchmarks. The experiments are organized around simulation performance, real-world performance, evaluation efficiency, and evaluation stability, reflecting the paper’s claim that a useful benchmark must be both diagnostically broad and operationally practical. RoboDojo’s simulation side uses heterogeneous parallelism in Isaac Sim to improve throughput by running different tasks and scenes concurrently, which supports faster model iteration. The real-world side evaluates policies under standardized physical conditions through RoboDojo-RealEval, making comparisons less dependent on ad hoc local lab setups. The resulting leaderboard and analysis are presented as a way to make policy limitations visible across capability dimensions and deployment settings, rather than compressing performance into an undifferentiated success rate.
One thing to remember
The key takeaway from RoboDojo is that evaluating generalist robot manipulation requires both breadth of capability testing and discipline in reproducibility. The benchmark’s anti-gaming and integrity protocols, public leaderboard, and non-profit governance by AI MMLab Club are intended to keep comparisons fair and independent as policies improve. By combining 42 simulation tasks, 18 real-world tasks, XPolicyLab integration, and RoboDojo-RealEval, the paper proposes an evaluation loop that can track progress from fast simulated diagnosis to physical-world validation. Its broader implication is that future robot manipulation research should report not only aggregate task success, but also how policies behave under memory demands, precision constraints, long-horizon sequencing, open-vocabulary instructions, and real deployment noise. RoboDojo positions benchmark infrastructure itself as a research contribution: a standardized testbed for finding the next bottlenecks in robust generalist manipulation.
