The team from Meta FAIR and the University of Washington jointly proposed a byte-level distillation method, which converts the token probability distribution of large teacher models into byte-level, enabling student models to learn byte-level probability distributions. The team introduced two conversion schemes: Marginalize-It and End-Of-Token, with the End-Of-Token scheme more effectively preserving the teacher's distribution. Experimental results show that token models learn faster in the early stages, while byte models exhibit stronger scaling potential. Using the End-Of-Token distillation scheme, the upper limit of downstream average accuracy is 4 percentage points higher than traditional token distillation, while significantly reducing storage pressure on data and teacher distributions. However, the training computation for this scheme is slightly higher, and a comparison of inference costs has not yet been completed.