Tsinghua Team Revisits On-Policy Distillation: Data Overload, Algorithm Starvation
2 day ago / Read about 0 minute
Author:小编   

Researchers from Tsinghua University and other institutions have conducted a study on On-Policy Distillation (OPD). By reducing the training set to just one problem, they discovered that the model could continuously improve its performance after training for several hundred steps. Training with a single problem could recover over 70% of the benefits of full-data training, and similar effects were observed across multiple tasks such as mathematics and programming, as well as with different models, showing insensitivity to the nature of the training problems. The study points out that OPD faces the issue of 'data overload, algorithm starvation': In terms of data, a single problem can generate a large number of states through repeated rollouts, with one problem covering 71.5% of the full state space. Just 16 problems with different semantics can achieve effects comparable to full-data training, indicating that the number of problems is not an appropriate metric for measuring effective supervision—state coverage is more critical. In terms of algorithms, the speed at which the model absorbs supervision gradually slows down as training progresses, and the size of the training set has minimal impact on the absorption rate. Even with a fixed set of states, the model still requires multiple updates to digest the supervision. Under a multi-teacher OPD (MOPD) setting, just 16 problems with different semantics in each domain can achieve the effects of full-data MOPD. Extreme experiments show that inputs without explicit task content can also provide effective supervision, with the key being whether the model can be guided into states where teachers can provide effective supervision. Additionally, compared to RLVR, OPD shows higher training gains, but both have ceilings limited by different factors. This research provides a new perspective on data selection for OPD, emphasizing the importance of whether problems can guide the model to explore unvisited states, whether teachers have teachable content, and the student's absorption speed. This state-centric perspective may also apply to other on-policy training paradigms.