Recently, major players in the AI industry, including DeepSeek, OpenAI, and Anthropic, have been acquiring NVIDIA GPUs while also embarking on the development and deployment of their own AI chips. The primary driver behind this shift is the inefficiency of general-purpose GPUs during the decoding phase of large-scale model inference, coupled with substantial data transfer costs that result in significant computing power wastage. Previously, due to the lack of a clear technological roadmap for large-scale models, these leading companies did not prioritize the development of dedicated inference chips. However, with the emergence of technological pathways such as deep reasoning and AI Agents, the demand for inference workloads has skyrocketed, highlighting the benefits of specialized inference chips. Google's early adoption of TPUs effectively tackled data transfer challenges through systolic array technology. Nowadays, each company has its own distinctive strategy for developing inference chips: Anthropic integrates systolic pipelines with AWS Trainium and performs reverse optimization on the compiler; Groq forgoes HBM memory in favor of SRAM to enhance efficiency; Alibaba embraces a flexible and adaptable architecture; OpenAI works with Broadcom to optimize KV Cache binding at the cluster level; DeepSeek is also set to deploy inference chips tailored to the unique technological features of its models. Looking forward, AI companies are expected to transition from adapting to GPUs to proactively selecting software and hardware solutions that best fit their needs, thereby liberating themselves from the limitations imposed by hardware vendors.
