Moore Threads has recently unveiled the open-source availability of its large-scale model training framework, Megatron-MUSA-Titan. Built upon Moore Threads' versatile, full-function GPU, this project is specifically designed to address the rigorous training requirements of trillion-parameter Mixture of Experts (MoE) models within expansive 10,000-GPU clusters. It caters to two fundamental scenarios: pre-training and reinforcement learning.

The framework equips developers with the necessary tools for training adaptation and system optimization tailored to the MUSA platform. It has been rigorously tested and continuously deployed within Moore Threads' 10,000-GPU clusters, undergoing extensive training validations. This sustained deployment has resulted in remarkable model computational efficiency and the ability to maintain stable, long-duration training sessions.

Moreover, the project fosters seamless collaboration with reinforcement learning frameworks like Slime and Miles. By integrating Megatron training with SGLang sample generation, it establishes a comprehensive training infrastructure that spans from trillion-parameter MoE pre-training to reinforcement learning, ensuring a holistic approach to model development.

From a technical standpoint, Megatron-MUSA-Titan boasts support for blockwise FP8 training. This feature effectively harnesses the low-precision computing prowess of Moore Threads' full-function GPU, significantly enhancing computational efficiency during large-scale training endeavors. Furthermore, by incorporating DeepEP alongside the ACE asynchronous communication engine, the framework expedites token distribution and result aggregation processes within MoE models. This integration facilitates the overlapping of expert parallelism and computation, thereby further elevating end-to-end training efficiency.