In today's era, marked by the swift evolution of large-scale artificial intelligence models, a pressing and cutting-edge challenge that garners widespread attention from both the academic and industrial sectors is how to guarantee the seamless and efficient functioning of Transformer models while simultaneously safeguarding user privacy. The data submitted by users frequently encompasses sensitive details pertaining to healthcare, finance, and voice recordings. Meanwhile, the parameters of these extensive models represent invaluable assets for enterprises.