Evaluation of 28 Agent Combinations: Highest Task Resolution Rate Reaches Only 14%
3 day ago / Read about 0 minute
Author:小编   

For large-scale software systems undergoing long-term evolution, existing code Agents, although proficient at invoking tools, modifying files, and running tests, still show obvious shortcomings in delivering complete functionality starting from new requirements. Researchers from Fudan University and other institutions have introduced the LoLBench evaluation benchmark, which places Agents in large-scale software systems averaging about 2.4 million lines of code to evaluate their complete long-horizon code task capabilities, from feature proposal to module delivery. Such tasks involve both perceptual complexity—identifying implicit constraints in requirements and cross-module dependencies, and deriving the scope of modifications and implementation specifications—and implementation complexity—completing correct cross-file modifications while avoiding breaking existing functionality. The evaluation covers 100 tasks across 29 systems, 8 languages, and 5 domains, verifying delivery correctness through a combination of newly added feature tests and regression tests, and optimizing test detection rates by constructing faulty implementations as mutants. Experiments on 28 frontier Agent combinations show that the current highest task resolution rate among Agents is only 14%, with a large amount of interaction and token consumption concentrated in the code localization phase, where code localization accuracy is low and coverage is incomplete. Even when additionally providing the modified file tree and API implementation specifications to reduce perceptual burden, the best combination's task resolution rate only increases to 34%, with 66% of tasks still unable to be completed, exposing the key breakpoint that Agents easily miss cross-module synchronized changes. The evaluation aims to promote the development of code Agent capabilities from mere code generation toward supporting the continuous evolution of large-scale software.

  • C114 Communication Network
  • Communication Home
7 X 24 Track global technological trends
Hot Topic