Shanghai AI Lab & Shanghai Jiao Tong University Propose a Novel Pre-training Task and Open-Source the First 8.9B Latent Space Large Model NCP-ArchPreview
4 day ago / Read about 0 minute
Author:小编   

The team from Shanghai Artificial Intelligence Laboratory and the LUMIA Lab at the School of Artificial Intelligence, Shanghai Jiao Tong University, has proposed a novel pre-training task—Next Concept Prediction (NCP)—and introduced the world's first discrete latent space foundation model with a scale of 8.9B, named NCP-ArchPreview. Pre-trained on the 5.73T Dolma-3 corpus, this model achieved the same final pre-training loss level as OLMo-3-7B using only 51.3% of the token budget, demonstrating a 1.95x improvement in equivalent convergence speed and a systematic 1.74x enhancement in computational Pareto efficiency. In terms of comprehensive performance on downstream tasks, NCP-ArchPreview leads by 2.45 points, with a nearly 6-point improvement in the GSM8K mathematical reasoning task. NCP-ArchPreview employs a three-stage latent space concept processing pipeline, constructing an ultra-large discrete concept representation space through product quantization. It achieves explicit future concept modeling using differentiable next concept prediction, a causal leakage prevention feedback mechanism, and hierarchical residual routing. Additionally, the model offers extra advantages: fine-tuning only 17M latent space parameters can surpass LoRA without forgetting, while improving training throughput and reducing GPU memory usage. Injecting the concept representations from NCP-ArchPreview into a draft model can increase the average acceptance length of speculative decoding by 4.17%, with a 7.59% improvement on code tasks. The team also shared pitfalls and solutions in their technical report, established a hundred-billion-scale proxy metric screening mechanism, and open-sourced all assets, including full-stage checkpoints and evaluation frameworks. This model breaks away from the word-by-word prediction paradigm, shifting towards latent space concept prediction, significantly improving training efficiency and opening up new directions for model fine-tuning and inference acceleration.