Watch, Simulate, and Experiment: Tsinghua's Asking the World Framework Lets AI Answer Physics Questions
16 hour ago / Read about 0 minute
Author:小编   

Building on their earlier PhysMind work, a team at Tsinghua University has proposed the Asking the World (ATW) framework. This framework requires no additional training or fine-tuning. Instead, it relies on a vision-language model to interpret questions and select operations, paired with physics tools that compute motion—enabling AI to autonomously construct runnable physics simulation worlds from videos. Through attribute queries, motion simulation, and conditional intervention, the system gathers evidence to answer questions about physical attributes, motion prediction, and counterfactual reasoning. The framework unfolds in two stages—world modeling and world probing—with the agent autonomously deciding both the modeling approach and the experimental content. The team also developed the supporting PolyWorld Engine, a physics simulation engine covering rigid bodies, soft bodies, cloth, ropes, fluids, and their coupled interactions, extending the applicable scenarios from rigid-body collisions to a wide range of materials and physical scenes. Tests show that with Gemini-3-Flash as the base model, ATW achieves overall accuracy of 80.82% and 70.56% on the two physics reasoning benchmarks CLEVRER and ContPhy, respectively—improvements of 46.50 and 28.10 percentage points over answering directly from video—and also demonstrates a clear advantage in real-world scene tests. The project homepage, code, and paper have all been made public.