A recent report found that the state of the U.S. dynamic random-access memory (DRAM) chip industry is in a “dire crisis.” This shortage is especially concentrated on high bandwidth memory (HBM), critical to the GPU and AI accelerator markets. This scarcity has led to skyrocketing costs and a rerouting of available supply from different uses. Firms such as Samsung are leading the way with innovative chip stacking technology. Industry experts are hopeful that this advancement could soon provide some much-needed relief to the market.
The demand for DRAM, most notably for use in AI data centers, has ballooned to all-time highs. This unprecedented surge in demand creates unimaginable pressure on the entire supply chain. At the same time, consumers and businesses are dealing with escalating costs. In this competitive and fast-paced technology landscape, HBM technology is more important than ever. It dramatically improves performance by efficiently packing memory and processors tightly together.
The Role of High Bandwidth Memory
Large amounts of high bandwidth memory is critical for keeping GPUs and other accelerators used in AI workloads fed. Each HBM chip can be made up of several vertically stacked DRAM dies, further quadrupling the memory bandwidth and density. The newly announced B300 model uses eight of these HBM chips, with each chip holding a stack of 12 DRAM dies.
HBMs benefit from being stacked vertically and mounted in an interposer on either side of the processor, providing a convenient and efficient design. This design approach greatly reduces latency by putting the HBMs within a millimeter of the GPU/AI accelerator. Additionally, HBMs are connected to these processors by up to 2,048 micrometer-scale connections, providing ultra-fast data transfers.
The breakthrough with chip stacking technology allowed a 16-high stack to become a reality. Samsung has surely proven its ability to manufacture this state-of-the-art HBM 2.5D configuration with hybrid bonding technologies. The current HBM4 standard only accommodates 16 stacked DRAM dies. Even so, Samsung wants to have us think we’re approaching the perfection point of stacking 20 dies.
“Relief will come from a combination of incremental capacity expansions by existing DRAM leaders, yield improvements in [advanced packaging], and a broader diversification of supply chains,” – Shawn DuBravac
The Impact of Demand on Supply Chains
The skyrocketing need for DRAM has sent a cascading effect throughout every sector that uses these vital components. Now that AI data centers are consuming unprecedented amounts of DRAM, other industries have to compete with data centers to secure dwindling supplies. A combination of limited competition and soaring demand have pushed prices up dramatically, impacting everything from consumer electronics to enterprise-level computing solutions.
Mina Kim, an industry analyst, pointed out two primary methods to address these supply issues: “There are two ways to address supply issues with DRAM: with innovation or with building more fabs.” This simple statement highlights the truth that not only is innovation required, but there must be a focus on building manufacturing capacity to feed the unending pipeline.
Given market conditions, it doesn’t seem likely that prices will come down anytime soon. Kim noted, “In general, economists find that prices come down much more slowly and reluctantly than they go up. DRAM today is unlikely to be an exception to this general observation, especially given the insatiable demand for compute.”
Future Developments in Memory Technology
Samsung is making bold moves to meet these challenges and face them head on. They announced plans to open a new foundry in Pyeongtaek, South Korea by 2028. This new facility will support increased production capacity and expansion to respond to greater global demand for innovative, advanced memory products.
With advancements in chip stacking technology, the idea of higher-density configurations starts to take shape. The industry is interested to see what happens next with HBM technology, especially as players look to find other processes that maximize yield and performance.
The introduction of new advanced packaging techniques is equally exciting for the potential to boost memory production efficiency. As our manufacturers continue to innovate their ways, they are increasing their capabilities. The more we anticipate our supply chains will level out and stabilize.