Mainstream test generation benchmarks that rely on single-reference evaluation tend to overestimate test quality. To tackle this problem, the team led by Liu Jiaheng at Nanjing University proposed a new benchmark called TestPrism. It includes 300 test tasks and 3,000 labeled diverse candidate implementations, with JSF as the core metric. A test must simultaneously fail on the original code, accept all correct implementations, and reject all incorrect implementations. Evaluation results show that the true reliability of tests generated by frontier Coding Agents is far lower than what single-reference evaluation suggests—the strongest configuration achieves a JSF of only 28.00%. The team further summarized three major categories and seven types of test generation error mechanisms, and proposed the TestHelix framework. Through heterogeneous synthesis, peer cross-validation, and recursive self-improvement, TestHelix raises JSF by 11.67 to 12.00 percentage points under the same inference budget, offering new ideas for more reliable test generation and evaluation.
