Tsinghua University, in Collaboration with Other Institutions, Unveils the World's Pioneering Benchmark for Streaming Interactive Video Evaluation
1 day ago / Read about 0 minute
Author:小编   

AI-driven real-time interactive video represents a burgeoning technological frontier; however, it grapples with challenges such as inconsistent definitions and a dearth of standardized evaluation metrics. Presently, the conventional tools employed for assessing video generation logic fall short in encompassing core competencies like instantaneous responses to continuous commands and ensuring state consistency. To bridge this gap, a consortium of leading universities and institutions, both domestic and international, has collectively launched StreamEval—the globe's inaugural benchmark tailored for streaming interactive video generation. This benchmark encompasses a comprehensive suite of 300 test scenarios spanning five distinct categories and establishes a quantifiable evaluation framework grounded in four key dimensions and ten specific indicators. Presently, all pertinent code, datasets, and scoring guidelines have been open-sourced, empowering developers to conduct independent retests and seamlessly integrate their proprietary models for evaluation. Evaluations of 38 models have been successfully completed, unveiling that each model harbors unique strengths, with the execution of continuous instructions emerging as a ubiquitous challenge confronting the industry. As a public evaluation framework serving the entire industry, StreamEval aspires to institute reproducible and continuously evolving unified evaluation standards for the real-time interactive video domain. It will be sustained over the long term, with monthly updated rankings, and developers can engage in the evaluation process in three straightforward steps by adhering to the public procedures.