The global memory chip market is experiencing an alltime unprecedented crisis. Demand for high-bandwidth memory (HBM), critical for graphics processing units (GPUs) and artificial intelligence (AI) accelerators, is skyrocketing. As companies ramp up efforts to build out AI infrastructure, they face a complex landscape shaped by both historic supply chain disruptions and advancements in chip technology. It’s expected that this current shortage will last through at least 2028, forcing manufacturers to think outside the box for potential solutions.
HBMs are vertically stacked and connected to processors using special I/O on each side. They are key to improving energy efficiency of AI data center performance. These memory chips are designed to function in perfect unison with the GPUs as one single entity. This integration allows for real-time data exchange between the two elements. The near-term demand for DRAM – especially DRAM that supports GPUs and other accelerators in AI applications – is huge. The industry has an urgent need to support an emergent AI hardware infrastructure. Concurrent with this market transformation, the traditional boom and bust cycles of the DRAM sector are meeting their match.
Industry Innovations and Challenges
In the past few months, Samsung has demonstrated its prowess at manufacturing a 16-high stack of DRAM dies. This accomplishment represents a historic step forward in the manufacture of HBM utilizing groundbreaking hybrid bonding technology. This development is very important. Fitting perfectly within the HBM4 standard, it accommodates integration of up to 16 DRAM dies stacked vertically. Today, most applications only leverage a handful of these potential dies. This shines a spotlight on the large gulf between what could be produced and what is actually being produced.
The recently introduced B300 model uses 8 HBM chips, each made up of 12 DRAM dies. These chips are typically deployed literally a millimeter away from the GPU and other AI accelerators. This tight coupling allows for optimized and high-speed data processing. The connection between HBM and the GPU supports up to 2,048 micrometer-scale connections. It’s the kind of precision that has accompanied greatly advanced levels of chip design today. The DRAM tower with HBM is about 750 micrometers in height, a testament to the intricacy of today’s semiconductor fabrication, with 3D IC technology.
Still, experts say jetting the full promise of 16 stacked dies depends on progress in the emerging field of high-speed chip stacking. With hybrid bonding, more aspirational goals are possible. We look forward to finally accomplishing this goal, so stay tuned…20 dies connected equals one happy TTP!
“Relief will come from a combination of incremental capacity expansions by existing DRAM leaders, yield improvements in [advanced packaging], and a broader diversification of supply chains,” – Shawn DuBravac
The Economic Landscape
The current supply-demand breach has caused HBM and other memory technologies prices to soar. Indeed, industry analysts have pointed out that the quickly rising prices are usually a great deal quicker than the equally swift drop once balance is returned.
“In general, economists find that prices come down much more slowly and reluctantly than they go up. DRAM today is unlikely to be an exception to this general observation, especially given the insatiable demand for compute,” – Mina Kim
“There are two ways to address supply issues with DRAM: with innovation or with building more fabs,” – Mina Kim
To this day, despite each of these concerted efforts, many industry executives remain doubtful about immediate relief from the present-day shortfalls. In other words, no relief at all—that’s the Intel CEO Lip-Bu Tan’s firm proclamation that “there’s no relief until 2028.” This timeline begs the question on what the long-term impact will be to companies that depend on these high-performance computing capabilities.
The Future of HBM Technology
Looking to the next wave of disruptive technologies and shifts in consumer demand, manufacturers are investing in advanced AI, robotics and other smart practices. Our expertise lies in improving and perfecting current production methods. Further, we take steps to diversify our supply chains to mitigate risks associated with reliance on particular regions or suppliers.
The confluence of technical innovation and market need has ushered in a time of dramatic change within the semiconductor industry. As companies invest in AI infrastructures, they must face the drawbacks brought on by memory shortages.
Continuous investments in HBM production technology provide a positive sign in the long term for addressing these supply constraints. Companies are working diligently to innovate their manufacturing processes, aiming to achieve higher yields and better efficiency as they push for increased output.