Search
Ant Group and vLLM Unveil Open-Source Humming: A High-Performance, Adaptable Quantized Inference Kernel
8 hour ago /
Read about 0 minute
Author：小编
On October 21st, Ant Group, in partnership with the vLLM community, announced the open-sourcing of Humming—a highly versatile and performance-optimized library for quantized inference operations. Humming is designed to support a broad array of GPUs and devices that are compatible with CUDA programming. Leveraging a unified architecture, it allows for seamless integration of various weight and activation formats within Dense and MoE models. This encompasses a comprehensive range of integer and floating-point types, bit widths, and scaling techniques. In performance evaluations across multiple standard workloads, Humming demonstrated a marked superiority over existing solutions, including vLLM Marlin, FlashInfer, and DeepGEMM.
-
C114 Communication Network -
Communication Home
7 X 24 Track global technological trends
