Tesla CEO Elon Musk recently took to the public stage to laud the latest research breakthroughs from Chinese AI firm Kimi. In a lighthearted twist, Kimi’s official account responded with a chuckle: “Your rockets are quite impressive too!” The technical report unveiled by the Kimi team introduced a groundbreaking Attention Residuals mechanism. This innovation marks a paradigm-shifting overhaul of the traditional residual connections, a staple in deep learning for nearly a decade, capturing the global spotlight.

By endowing each layer with an “intelligent filter,” this mechanism empowers the model to dynamically sift through and extract valuable information from preceding layers. This not only boosts transmission efficiency but also tackles challenges like the dilution of shallow information and inefficient training, which have long plagued traditional residual connections. Real-world tests reveal that the training efficiency of a 48B parameter model has surged by 1.25 times. Moreover, its scores in scientific reasoning and math problem-solving have climbed by 7.5% and 3.6%, respectively.