Achieving Acceleration Without Compromising Intelligence: Xiaohongshu's DoSA Cuts First Token Latency in Half
8 hour ago / Read about 0 minute
Author:小编   

The Dots team at Xiaohongshu has independently crafted the DoSA sparse attention mechanism. By harnessing the power of score upper and lower bounds, this innovative scheme swiftly tackles the indexer bottleneck that often plagues long-context inference, all the while maintaining the integrity of the original Top-K selection outcomes. It achieves this by pre-filtering around 97% of candidate tokens. The DoSA mechanism establishes a lower bound for the selection threshold and an upper bound for candidate scores, thereby eliminating candidates that are unlikely to make it into the Top-K. It then conducts precise scoring on the remaining candidates using full index heads exclusively. Additionally, it refines the GPU execution process, sidestepping the scattered computational overhead that can slow things down. Test outcomes reveal that, at a context length of roughly 1 million tokens, the first token latency for Dots3-Note-preview, GLM-5.2, and DeepSeek-V3.2 is slashed by approximately 37%, 46%, and 48%, respectively, when pitted against their respective SGLang baselines. The indexer's acceleration prowess far surpasses that of existing similar solutions.