Baidu Open-Sources ERNIE 5.5 Flash Base, Achieving SOTA Across Multiple Benchmarks for Models of the Same Size
7 hour ago / Read about 0 minute
Author:小编   

Baidu has officially open-sourced its latest foundation model, ERNIE 5.5 Flash Base. Built on an MoE architecture, the model features a total of 280 billion parameters, activates 14 billion parameters per token, and was pretrained on 33.5 trillion tokens. Its pretraining context window reaches up to 128K, while the fine-tuning and inference stages support contexts of up to 1 million tokens—achieving state-of-the-art (SOTA) performance among models of the same size across multiple benchmark categories. With efficiency as its central design principle, the model implements systematic optimizations and a series of algorithmic innovations in key areas such as attention mechanisms, expert routing, and training mechanisms. It also optimizes the distributed Muon optimizer and introduces FlashMask-4, an attention operator designed for large head dimensions, thereby improving communication, memory utilization, and computational efficiency in large-scale training. As a foundation model for developers and enterprises, ERNIE 5.5 Flash Base offers ample room for fine-tuning and customization, enabling the development of specialized models and applications tailored to diverse scenarios. By lowering development and training costs and reducing technical barriers, it provides the industry with a more extensible technical foundation, driving capability exploration and application innovation in large model technology.

  • C114 Communication Network
  • Communication Home
7 X 24 Track global technological trends
Hot Topic