WorldOracle, the world's first action-driven world model evaluation platform benchmarked against real-world execution, has been officially rolled out and is now welcoming evaluation submissions from developers. Developed through a collaborative effort by researchers from ZhiYuan, NVIDIA, Princeton University, the University of Texas at Austin, and the University of Toronto, this platform primarily assesses the accuracy of world models in predicting the outcomes of robot action execution. In contrast to traditional evaluation methods that emphasize the quality of video generation, this platform leverages real-world robot operation data as a reference point. By juxtaposing the predicted videos generated by the model with the actual execution processes, it underscores the importance of testing the model's predictive prowess in physical interaction scenarios. Presently, the platform encompasses 335 real operation trajectories, covering a range of tasks including fruit grasping, towel hanging, water pouring, and drawer opening/closing. Notably, around 40% of the interaction clips feature failed or suboptimal operations, providing a comprehensive and realistic testing ground.
