PingWest, March 17th News - Moonshot AI has recently introduced an innovative architecture called Attention Residuals (AttnRes), aimed at improving the information processing methods of large language models based on Transformer. In traditional residual connections, the outputs of each layer are superimposed with equal weights, leading to information blurring. AttnRes introduces a deep attention mechanism, enabling network layers to dynamically select and weightedly combine information from previous layers. This method treats model depth as a sequence dimension, allowing each layer to actively retrieve historical features rather than passively receiving mixed signals. It effectively addresses the issues of hidden state redundancy and lack of selective access in deep networks, significantly enhancing the stability and efficiency of models in long-context reasoning. As a technological breakthrough behind the Kimi series of models, AttnRes reflects the trend of extending attention mechanisms to the hierarchical structure of networks. Moonshot AI continues to drive the development of large models through architectural innovations, with its trillion-parameter mixture-of-experts system already applied to complex reasoning tasks. The introduction of AttnRes signifies that even the most fundamental residual components are still evolving towards greater efficiency and adaptability, laying a theoretical foundation for building the next generation of high-performance AI systems.