Recently, two Chinese AI companies have taken the initiative to open-source inference systems that are finely tuned for their proprietary models. This move signifies a strategic shift in optimization efforts, extending beyond the models themselves to encompass the inference systems that power them. Among these pioneering companies, StartLux has unveiled a dedicated inference engine, StartLux-Prism, which boasts four core optimizations meticulously crafted to cater to the unique characteristics of decision-making models. Unlike models that generate lengthy texts token by token, decision-making models focus solely on calculating probabilities for a limited set of candidate options. The four optimizations are as follows: 1) Prism calculates only the output scores of candidate answers, thereby minimizing output layer overhead; 2) It organizes multi-question tasks that share the same input material into a 'forest' structure, enabling the reuse of public context computation results and enhancing efficiency; 3) By leveraging INT8 Tensor Core and INT32 accumulation for deterministic computation, Prism ensures bit-level consistency of output probabilities across different concurrency scenarios; 4) It facilitates the sharing of GPU resources between long tasks that generate thinking processes and short tasks that yield immediate results, thus preventing long tasks from obstructing the processing of short requests. Tests have demonstrated that when executing the StartLux-Decision v1.1 27B model on a single NVIDIA H200 GPU, Prism's inference speed surpasses that of the original vLLM solution by a factor of 1.2 to 5.3. This acceleration is particularly pronounced in specific scenarios, where it enhances throughput by approximately 17% under high concurrency conditions. Moreover, the accuracy of the quantized model remains largely on par with its original precision. Prism also supports combined deployment with both tensor parallelism and data parallelism, offering flexibility and scalability. Dedicated inference optimizations tailored to specific models and scenarios are emerging as a new competitive edge in the AI landscape.
