Early this morning, DeepSeek made headlines by open-sourcing a cutting-edge architectural module named 'Engram,' along with the release of an accompanying technical paper. Among the distinguished authors is Liang Wenfeng. The Engram module introduces a scalable, lookup-based memory structure, ushering in a fresh dimension of sparsity for large-scale models.

At present, mainstream large models grapple with structural inefficiencies, particularly when dealing with 'lookup-table-style' memory and intricate reasoning computations. Leveraging modern hash N-Gram embeddings, Engram achieves an O(1) lookup-based memory system, ensuring stable retrieval costs. Moreover, it offers 'conditional memory,' strategically positioned in the initial layers of the model to handle 'pattern reconstruction' tasks.

In experiments conducted with a 27-billion-parameter model, under conditions of equal parameters and computational power, the model exhibited marked performance enhancements across a variety of tasks. Discussions surrounding these findings suggest that Engram diminishes the necessity for early-layer models to engage in static pattern reconstruction. Some developers have highlighted that this architecture liberates large-scale static memory from the constraints of GPU storage, while also keeping inference overhead minimal.

Several industry observers speculate that Engram might serve as the cornerstone technology for DeepSeek's upcoming 'V4' model.