Pre-training Transformers Without Backpropagation: Achievement from a Company Backed by Jeff Dean
2 day ago / Read about 0 minute
Author:小编   

Q Labs has proposed a zero-order optimization method called Dust, which enables pre-training of Transformers without the need for backpropagation. This method transfers weight perturbations from traditional evolutionary strategies to the activation space, independently adding Gaussian noise to each token to construct a virtual population. It evaluates a large number of virtual individuals in parallel through a single forward pass and estimates gradients using a credit assignment mechanism with a decay coefficient. In experiments with models under 243 million parameters, Dust outperformed similar zero-order methods and approached or even surpassed backpropagation under certain settings in large-scale populations. However, Dust currently has significantly higher computational costs than backpropagation and has only been validated in small-scale experiments. Third-party reproduction results show that backpropagation remains clearly superior under conventional training conditions. The research team is committed to exploring efficient utilization of computational resources and overcoming the differentiability constraints of backpropagation. Subsequent optimizations by the relevant community have achieved substantial speedups in small-scale benchmarks, but the method's scalability in large models and large-scale training scenarios remains to be verified.