The 3B Model Attains a 94.5% Success Rate: Token-Level Rewards Pave the Way for Training Long-Chain Agents
2 day ago / Read about 0 minute
Author:小编   

When multi-round agent tasks are configured with endpoint rewards only, the challenge of temporal credit assignment emerges. This makes it challenging to pinpoint which step along the trajectory requires reinforcement. The prevailing mainstream strategy involves employing a model to score tokens in unskilled trajectories. However, the confidence of the "teacher" model may lean more towards linguistic fluency rather than actions that are crucial to the task. The team from the University of Science and Technology of China and Alibaba has introduced the ADRS method. This method converts privileged skills acquired during training into token-level credit assignments and incorporates them into the reinforcement learning framework. Through a three-step training process, ADRS leverages a three-tier credit construction mechanism to tackle issues such as the incomparability of teacher scores across different steps, the potential irrelevance of confidence levels, and the circumvention of group relative advantages through independent distillation losses. Notably, this method does not modify the underlying RL algorithm and requires no additional skills during the inference phase. In tests conducted in environments such as ALFWorld, WebShop, and search-based QA settings, ADRS substantially enhanced model success rates. It accurately reinforced tokens critical to decision-making, exhibited high data efficiency, and demonstrated greater advantages in long interaction chains, while also delineating the method's limitations.