Why your next laptop will have an NPU, just like your phone

If you’re considering buying a new laptop, you’ve no doubt noticed that they’re increasingly boasting NPU capabilities that sound awfully similar to the hardware we’ve seen in top smartphones for several years. . The driving factor is the drive for laptops to catch up with mobile AI capabilities and equip them with advanced AI features such as Microsoft’s Copilot, which can run securely on devices without the need for an internet connection. So here’s everything you need to know about NPUs, why your next laptop might have them, and whether or not you should buy one.

What is NPU?

NPU stands for Neural Processing Unit. NPUs focus on running mathematical functions associated with neural network/machine learning/AI tasks. While these may be standalone chips, they are increasingly integrated directly into a system-on-chip (SoC) alongside the more familiar CPU and GPU components.

NPUs come in a variety of shapes and sizes, and are often called slightly differently depending on the chip designer. You can already find different models scattered across the smartphone landscape. Qualcomm has Snapdragon Hexagon in its processors, Google has its TPU for the cloud as well as its Tensor mobile chips, and Samsung has its own implementation for Exynos.

This idea is now being developed in the notebook and PC space as well. For example, there is a Neural Engine in the latest Apple M4, Qualcomm Hexagon features in the Snapdragon X Elite platform, and AMD and Intel have started integrating NPUs into their latest chipsets. While not quite the same, NVIDIA GPUs blur the lines, given their impressive number crunching abilities. NPUs are increasingly everywhere.

As we mentioned, NPUs are purpose-built to handle machine learning workloads (along with other math-intensive tasks). In layman’s terms, the NPU is a very useful, perhaps even necessary, component for running AI on-device rather than in the cloud. As you’ve no doubt noticed, AI seems to be everywhere these days, and building support directly into products is a key step in that journey.

Much of today’s AI processing is done in the cloud, but this is not ideal for several reasons. The first is latency and network requirements; you can’t access the tools when you’re offline, or you might have to wait for long peak processing times. Sending data over the internet is also less secure, which is a very important factor when using AI that has access to your personal information, such as Microsoft’s Recall.

Simply put, running on the device is preferred. However, AI tasks are very computationally intensive and do not run well on traditional hardware. You may have noticed this if you tried to generate images with Stable Diffusion on your laptop. It can be painfully slow for more advanced tasks, although the CPUs can run a number of “simpler” AI tasks just fine.

The solution is to use dedicated hardware to speed up these advanced tasks. You can read more about what NPUs do later in this article, but the TLDR is that they run AI tasks faster and more efficiently than your CPU can do alone. Their performance is often quoted in trillions of operations per second (TOPS), but that’s not a very useful metric because it doesn’t tell you exactly what each operation is doing. Instead, it’s often better to look for numbers that tell you how fast it takes to process tokens for large models.

Speaking of TOPS, NPUs for smartphones and early laptops are rated in the TOPS tens. Broadly speaking, this means they can speed up basic AI tasks like camera object detection for bokeh blur or text summarization. If you want to run a large language model or use generative AI for fast media production, you’ll want a more powerful accelerator/GPU in the hundreds or thousands of TOPS.

A neural processing unit is quite different from a central processing unit because of the type of workload it is designed for. A typical processor in your laptop or smartphone is fairly versatile for a wide variety of applications, supporting wide instruction sets (the functions it can perform), different ways of storing and recalling functions (to speed up repetitive loops), and large out-of-order execution windows (so they can continue at work instead of waiting).

However, machine learning tasks are different and don’t require as much flexibility. For starters, they are much more math intensive, often requiring repetitive computationally expensive instructions such as matrix multiplication and very fast access to large pools of memory. They also often work with unusual data formats, such as sixteen, eight, or even four-bit integers. In comparison, your typical CPU is built around 64-bit integers and floating-point math (often with additional instructions).

Creating an NPU dedicated to mass parallel computation of these specific functions results in faster performance and less energy wasted on idle functions that are not useful for the task at hand. However, not all NPUs are created equal. Even beyond their sheer number-crunching capabilities, they can be built to support different types and integer operations, meaning that some NPUs are better at working on certain models. For example, some smartphone NPUs run on INT8 or even INT4 formats to save power, but you get better accuracy with the more advanced but power-hungry FP16 model. If you need really advanced calculations, dedicated GPUs and external accelerators are still more powerful and format-variable than integrated NPUs.

As a backup, CPUs can run machine learning tasks, but they are often much slower. Modern CPUs from Arm, Apple, Intel, and AMD support the necessary math instructions and some minor quantization levels. Their bottleneck is often just how many of these functions they can run in parallel and how fast they can move data in and out of memory, which is what NPUs are specifically designed to do.

While far from essential, especially if you’re not interested in the AI ​​trend, NPUs are required for some of the latest features you’ll find in the mobile and PC space.

For example, Microsoft’s Copilot Plus specifies a 40TOPS NPU as the minimum requirement you’ll need to use Windows Recall. Unfortunately, Intel’s Meteor Lake and AMD Ryzen 8000 chips found in current laptops (at the time of writing) do not meet this requirement. However, AMD’s newly announced Stix Point Ryzen chips are compatible. You won’t have to wait long for an x64 alternative to Arm-based Snapdragon X Elite laptops, as Stix Point-powered laptops are expected in H1 2024.

Popular PC-grade tools like Audacity, DaVinci Resolve, Zoom, and many others are increasingly experimenting with more sophisticated AI capabilities on the device. While these features aren’t necessary for basic tasks, they’re becoming increasingly popular, and AI capabilities should be considered in your next purchase if you use these tools regularly.

When it comes to smartphones, features and options vary a bit more from brand to brand. For example, Samsung Galaxy AI only runs on its powerful Galaxy S flagship phones. It didn’t bring features like a chat assistant or an interpreter to the affordable Galaxy A55, probably because it lacks the necessary processing power. This means that some of Samsung’s features also run in the cloud, but they likely won’t be funded by more affordable purchases. Speaking of, Google is equally good when it comes to feature consistency. You’ll find the best of Google’s AI additions like Video Boost on the Pixel 8 Pro – but the Pixel 8 and even the affordable 8a use many of the same AI tools.

Finally, there’s AI and NPUs are key to leveraging features on devices that can’t run on older hardware. That said, we’re still in the early days of AI workloads, especially in the notebook space. Software requirements and hardware options will only grow in the coming years. In that sense, it won’t hurt to wait for the dust to settle before jumping in.