The Chinese University of Hong Kong (Shenzhen) and Microsoft Research have jointly proposed ConvexLLM, a novel architecture for unified multimodal modeling. Leveraging the innovative C-Gate design, this architecture maps audio information—such as speech—into the convex hull of the original text word vectors of large language models. It generates audio representations as non-negative weighted combinations of frozen word vectors, eliminating the need for prior transcription into text. This allows sound to directly harness the knowledge and reasoning capabilities accumulated during large-model pretraining. In multiple benchmark tests, including VoiceBench, this architecture achieves leading performance among open-source models of the same scale, surpassing the strongest baseline by 8.8 percentage points on BBH tasks. It can also simultaneously adapt to multiple types of audio understanding tasks, such as environmental sounds and music, offering a new technical pathway for unified modeling of speech and language.
