DeepSeek has proudly declared the comprehensive open-sourcing of its proprietary inference framework, DeepInfer, along with its Token encoding/decoding module, deepseek-recipe. These innovative tools are designed to seamlessly integrate with both NVIDIA and Huawei Ascend platforms, with the primary goal of empowering developers and enterprises to deploy large-scale models with enhanced efficiency. This move is expected to significantly slash the computational expenses and energy footprint associated with AI services.

Leveraging the capabilities of the DeepInfer framework, DeepSeek V4.1 Flash is capable of processing an impressive 1.55 trillion Tokens daily within a single cluster. When factoring in off-peak electricity rates, this translates to a remarkable revenue generation of $38 per card per hour. DeepInfer achieves this heightened inference efficiency through a suite of cutting-edge technologies. These include the separation of Prefill and Decode processes for streamlined deployment, dynamic resource allocation, tiered KVCache retention for up to 7 days to facilitate reuse, speculative decoding with dynamically adjustable validation ranges, and asynchronous scheduling. Furthermore, the framework supports structured output functionalities.

Simultaneously, the open-sourced deepseek-recipe is engineered to be compatible with mainstream interfaces. It boosts long-text processing efficiency through a variety of mechanisms and guarantees alignment between third-party deployment logic and official services. Nevertheless, it's worth noting that code segments that are tightly intertwined with internal infrastructure, such as KVCache management and load balancing, have not been fully open-sourced at this juncture.