As AI advances at a rapid pace, the hardware behind it is experiencing an industrial seismic shift. Neural Processing Units (NPUs) are quickly becoming the bedrock of this shift, offering AI models quicker and more optimized performance. As technology continues to advance, future NPUs will be capable of processing even more tokens per second, making the browsing experience even smoother and more seamless. Competitors such as Qualcomm, AMD and Intel are in cutthroat competition with each other, pushing NPU performance to the limits. Many currently available laptops, especially those older than a year, will fall short of being able to stay up-to-date with these advances.
With NPUs now fully-integrated into laptop architecture, Windows is uniquely-positioned to execute even the most resource-intensive AI tasks efficiently. Windows ML runtime intelligently schedules work to the best hardware, from CPU to GPU to NPU, in real-time. This smart distribution ensures the best performance on your apps. This intelligence driven division of labor is essential to unlock the potential of today’s AI apps.
The Promise of NPUs
NPUs are purpose-built chips specialized for AI workloads, making them far more efficient than traditional CPUs. And they are really good at processing huge amounts of data, fast and efficiently. Recent developments indicate NPUs are close to providing performance ratings in the thousands of TOPS. So that works out to trillions of operations per second! AMD’s Ryzen AI Max packs an impressive NPU with a 50 TOPS rating. This newfound capability demonstrates its tremendous potential to accelerate AI processing.
Steven Bathiche, one of AI hardware’s preeminent powers, expands on what NPUs are actually capable of. He explains, “With the NPU, the entire structure is really designed around the data type of tensors [a multidimensional array of numbers].” With this design, NPUs are able to handle more intricate matrix calculations. They do all of these AI tasks much faster and more efficiently than CPUs.
Even for the NPU manufacturers themselves, competition is beginning to get fierce. Qualcomm’s Snapdragon X chip features an NPU that powers Microsoft’s Copilot+ features. As far as competitive systems go, AMD and Intel are closing the gap quickly. Their NPUs beat Snapdragon’s performance, driving up to 40-50 TOPS. These developments represent an unmistakable push in the direction of improved performance and efficiency in AI processing.
Challenges for Older Laptops
Despite these limitations, NPUs continue to have great potential and promise for the future. Unfortunately, most laptops available today cannot keep up with the expectations of sophisticated AI models. The average laptop that’s a year or older can barely run the more useful AI models locally. The options on the table are shockingly scant. This reality illuminates the need for hardware improvements. If developers need to harness the full power of new AI technologies, users need to invest and prepare to embrace them.
Another limiting factor for NPU performance is a common divided memory architecture found in many current PCs. Joe Macri highlights this issue by noting, “When I want to share data between our [CPU] and GPU, I’ve got to take the data out of my memory, slide it across the PCI Express bus, put it in the GPU memory.” Yet this process inevitably creates bottlenecks that significantly slow performance.
Beyond that, for companies willing to purely pursue the AI edge, NPUs represent a significant investment that needs to be balanced against investment in longstanding CPU and GPU technologies. Mike Clark points out that while specializing in AI is essential, maintaining traditional capabilities is equally important: “We must be good at low latency, at handling smaller data types, at branching code—traditional workloads.” This two-pronged approach keeps devices diverse and flexible to do a wide variety of computing and student needs.
The Road Ahead for AI Hardware
On the road ahead, the next few years are expected to bring even more groundbreaking innovations in NPU technology. These experts believe that NPUs that put thousands of TOPS inside laptops will soon be a reality, changing the way laptops tackle AI workloads. Nvidia’s new GeForce RTX 5090 demonstrates an astounding AI performance rating of 3,352 TOPS. As this remarkable number shows, it’s hard to miss the clear signal that’s being sent about how the market is progressing.
NPU technology is progressing by leaps and bounds. Manufacturers must now not only deliver chips that perform well in AI processing, but meet the thin and light designs required by today’s laptops. Mahesh Subramony emphasizes this point: “We have to be very deliberate in how we design our [system-on-a-chip] to ensure that a larger [SoC] can perform to our requirements in a thin and light form factor.” Getting this balance right will be imperative as consumer appetite for portable but powerful devices only deepens.
Even with all that hardware, software optimization will be equally important in order to get the most from the NPUs’ broad capabilities. Rakesh Anigundi notes that users will increasingly desire applications that can run continuously, such as AI personal assistants that remain active and responsive. “You’ll want to be running this for a longer period of time,” he explains.
