Shaping the Future of AI Chips: Custom HBM and Advanced Packaging Lead the Way
From innovations in CoWoS and Hybrid Bonding to the shift towards 3D-IC and HBM integration, the race to redefine high-performance memory and computing intensifies.
With the growing demand for advanced packaging, we have seen companies like Marvell and Broadcom launching their own advanced packaging platforms. For example:
Broadcom’s 3.5DXDSiP utilizes TSMC’s CoWoS-L packaging technology, enabling a packaging size approximately 5.5 times larger than the reticle size, with a total area of 4,719 mm². It integrates logic ICs, up to 12 HBM3/HBM4 stacks, and additional I/O chips.
To maximize performance, Broadcom recommends decomposing the compute chip design and employing Hybrid Copper Bonding (HCB) in a face-to-face (F2F) configuration to stack one logic chip on top of another. This approach utilizes bumpless hybrid bonding, directly connecting the metal layers of the upper and lower silicon dies. Compared to face-to-back (F2B) configurations that rely on through-silicon vias (TSVs), this method represents a key advantage of Broadcom’s 3DXDSiP platform.
Marvell highlights that with its proprietary D2D (die-to-die) I/O, it can accommodate 25% more logic chips in custom XPU designs and install 33% more CHBM memory packages alongside the compute chips, increasing the available DRAM capacity for processors. The company anticipates a 70% reduction in memory interface power consumption.
As advanced packaging technologies evolve, HBM (High Bandwidth Memory) has emerged as a pivotal component for AI and data-intensive workloads. A stable and reliable supply of HBM is now a critical factor for gaining a competitive edge in AI chip development.
The Current HBM Landscape
SK Hynix currently dominates the high-end HBM market, supplying leading players with state-of-the-art memory solutions. However, with demand surging, other companies aiming to enter the HBM market are developing proprietary technologies to create “HBM-like” products
MR-MUF & Its New Materials: The Missing Pieces to the Heat Control Puzzle
As SK hynix was developing HBM2E, the third generation of HBM, controlling heat became a major focal point for improvement. Even when TC-NCF was being recognized as a packaging solution suitable for densely stacked products, SK hynix challenged the status quo and strove to develop a new packaging technology offering improved heat dissipation. After countless tests and trials, the company unveiled its new packaging technology MR-MUF in 2019 which would change the future of the HBM market.
Emerging Players and Innovations
For example:
Nanya Technology: Through its subsidiary Patch Technology, Nanya has developed HBLL-RAM, a high-bandwidth memory solution employing a 3D-stacked architecture.
Winbond and Powerchip/ApMemory: Both have introduced solutions like Cube Memory, targeting niche and emerging HBM applications.
Source: Winbond CUBE enhances the performance of front-end 3D structures such as chip on wafer (CoW) and wafer on wafer (WoW), as well as back-end 2.5D/3D chip on Si-interposer on substrate and fan-out solutions. Designed to meet the growing demands of edge AI computing devices, it is compatible with memory density from 256Mb to 8Gb with a single die, and it can also be 3D stacked to enhance bandwidth while reducing data transfer power consumption.
Source: Winbond
These innovations reflect how new entrants are tackling the HBM challenge by exploring alternatives to conventional processes.
Key Challenges in HBM Development
To compete in the HBM domain, companies must overcome significant technical hurdles, including:
TSV (Through-Silicon Via): Essential for enabling vertical data pathways in 3D-stacked memory.
TCB (Thermal Compression Bonding): A critical process for attaching stacked dies with high precision.
Hybrid Bonding Technologies: Required for reliable die-to-die and die-to-wafer connections.
HBM Base Die Development: Involves designing a robust and efficient base die for memory stacks.
These technologies demand substantial R&D investment, expertise, and collaboration across the supply chain.
Future Outlook
The race to innovate in HBM and advanced packaging will continue to intensify as AI and high-performance computing drive demand. Companies entering this space must not only focus on technology development but also ensure a secure and scalable supply chain.
In subsequent discussions, I can delve deeper into specific technologies, market dynamics, and the strategies needed to succeed in the rapidly evolving HBM landscape.