The Tencent Hunyuan team, along with researchers from Tsinghua University and Peking University, has introduced the WebCraftBench evaluation framework. This innovative system incorporates software testing methods into the evaluation of AI-generated webpages. It enables agents to interact with webpages practically, identify shortcomings through code coverage, and score them based on three dimensions: aesthetics, usability, and requirement compliance. WebCraftBench conducted interactive exploration and scoring on 6,273 generated applications from 17 cutting-edge models, based on 369 real user requirements and 5,088 acceptance criteria. In a comparison of 197 sets of reviewed applications, its consistency rate with human preference judgments reached as high as 85.3%. The core process of this evaluation includes code instrumentation, agent exploration and operation, organizing operation records into state transition diagrams, and scoring across different dimensions. The evaluation results revealed that no model ranked first in all three dimensions. Aesthetics and usability showed low correlation at the individual application level, and a larger score difference correlated with a higher consistency rate with human preferences. Meanwhile, changing the scoring judges had little impact on the overall rankings, and coverage feedback helped improve exploration coverage. Additionally, the study provided key insights into webpage generation evaluation, including the need to evaluate along user operation paths, focus on exploration coverage, separate exploration from scoring, and maintain a rational perspective on leaderboards.
