At present, the artificial intelligence (AI) sector is undergoing a pivotal shift from compute-centric to memory-centric paradigms. Memory resource scarcity has emerged as the primary bottleneck in technological advancement, with high-performance memory components constituting up to 75% of the total cost of AI servers. To tackle this pressing issue, Google has implemented a multifaceted hardware-software optimization strategy. On the hardware front, the company is repurposing memory modules from decommissioned servers, enabling DDR4 memory to be integrated with TPU8i chips—which are designed to utilize DDR5—through the use of adapters and compatible interfaces. This approach helps mitigate the strain on DDR5 supply chains. Despite the TPU8i's adoption of a hierarchical storage architecture, DDR5 shortages have compelled some AI training tasks to revert to older-generation memory solutions. On the software side, Google has undertaken a comprehensive refactoring of its foundational function libraries, refined model architectures, and optimized KV cache compression algorithms. These efforts have collectively reduced memory consumption per computational unit by more than 30%.

This memory crisis has set off a ripple effect throughout the industry, prompting numerous cloud service providers to extend the operational lifespan of DDR4 equipment. Simultaneously, memory manufacturers are ramping up production to meet demand. Google's innovative strategies are increasingly being adopted by other enterprises seeking to navigate the memory shortage landscape.