NPU vs GPU for AI: Which Is Better for Laptops and Local LLMs?

On this page

1. 01 · NPU vs GPU in One Minute
2. 02 · What Is an NPU in a Laptop?
3. 03 · What NPU TOPS Means, and Why It Misleads
4. 04 · NPU vs CPU vs GPU Inside an AI PC
5. 05 · NPU vs GPU for Local LLMs
6. 06 · Software: How Apps Reach the NPU and GPU
7. 07 · Power, Battery and Heat
8. 08 · When to Use the Cloud Instead
9. 09 · Key Takeaways
10. 10 · Frequently Asked Questions

01 · NPU vs GPU in One Minute

AI PCs, Copilot+ PCs and running models on your own laptop

Two AI engines, two very different jobs

The short answer to NPU vs GPU is that they are built for different kinds of AI work. A neural processing unit (NPU) is a small, low-power AI accelerator built into modern laptop and phone chips. It is designed to run AI features efficiently for hours, such as blurring your background on a video call or generating live captions, without draining the battery. A graphics processing unit (GPU) is a much larger parallel engine. It burns more power, but it has far more raw throughput and, crucially for language models, access to more memory bandwidth.

So the NPU is usually the better choice for small, always-on AI features, and the GPU is usually the better choice for heavy AI work such as running a local large language model (LLM) or generating images. Most AI PCs have both, plus a CPU that runs everything else, and which engine a task lands on explains most of what you see in battery life, speed and heat.

02 · What Is an NPU in a Laptop?

Answer: An NPU, or neural processing unit, is a dedicated block on a laptop's main chip that runs neural network math, mostly low-precision matrix multiplications, using far less power than a CPU or GPU. It exists so AI features can run continuously on battery.

What is NPU in laptop terms: a specialist, not a replacement

Modern laptop processors are a system-on-chip (SoC): CPU cores, an integrated GPU and, increasingly, an NPU share one piece of silicon and one pool of memory. The NPU is the newest member. It does one family of operations very well: the multiply-and-add arithmetic at the heart of neural networks, usually at low precision such as 8-bit integers (INT8). Because it skips the general-purpose machinery a CPU needs and the graphics machinery a GPU carries, it can do that math at a fraction of the power.

Phones had NPUs first, because a phone battery cannot afford to wake a big GPU for every photo. Laptops followed: always-running features such as noise suppression, live captions and on-device search need an engine that sips power.

The NPU is not a faster GPU. It supports a narrower set of operations, and software must be prepared specifically for it, so an NPU can sit idle while your favourite app uses only the GPU.

03 · What NPU TOPS Means, and Why It Misleads

Answer: TOPS stands for trillions of operations per second. Vendors use it to rate NPU peak throughput, usually at INT8 precision. It is a useful rough guide, but TOPS figures from different vendors are not directly comparable, and they say nothing about memory or software support.

If you have searched what is NPU TOPS, the idea is simple: the number counts how many basic math operations the NPU can perform each second at its peak. A 45 TOPS NPU can, in theory, perform 45 trillion operations per second. The number became famous because Microsoft set a bar: to qualify as a Copilot+ PC, a Windows laptop needs an NPU rated at 40 TOPS or more, plus at least 16 GB of RAM and 256 GB of storage.

Treat the table as a starting point, not a leaderboard. Each vendor measures peak throughput under its own assumptions about precision, sparsity and clock speed. Real performance also depends on memory, on whether the model's operations are supported, and on how mature the vendor's software is.

04 · NPU vs CPU vs GPU Inside an AI PC

Answer: In an AI PC, the CPU runs the operating system and orchestrates work, the NPU handles efficient, sustained AI features, and the GPU, integrated or discrete, handles heavy parallel AI. Large jobs that no laptop can hold go to cloud GPUs.

The best way to understand an AI PC is to follow the workloads. The figure shows a typical laptop SoC with CPU cores, an integrated GPU and an NPU sharing system memory. Some laptops add a discrete GPU with its own video memory (VRAM), connected over PCIe. On the right, an OCI region takes jobs that outgrow the laptop. Each coloured arrow shows where a type of work usually lands.

Notice the memory story. On the SoC, CPU, GPU and NPU all draw from the same RAM, so a model loaded once can be used by any of them. A discrete GPU is faster but works from its own VRAM, so data must be copied across the PCIe link and the model must fit in that VRAM.

NPU vs CPU: why not just use the processor?

A CPU can run small AI models, and for occasional tasks that is fine. But its few powerful cores are tuned for serial logic and are busy running your operating system and apps. Continuous AI work on the CPU costs power and responsiveness, so the NPU takes it over.

05 · NPU vs GPU for Local LLMs

Answer: For local LLMs, the GPU usually wins. Generating text is limited mainly by memory bandwidth and capacity, not raw TOPS, and GPUs, especially discrete ones with fast VRAM, have more of both. The NPU is enough for small models in apps built to use it.

NPU vs GPU for local LLM chat: memory is the bottleneck

A local model means no per-token fees and no data leaving the device. But an LLM is mostly a large pile of weights, and to generate each token the chip reads essentially all of them from memory. Token speed therefore depends more on memory speed than on operations per second.

Some simple arithmetic shows why. A model with 7 billion parameters stored at 4-bit precision needs roughly 3.5 GB just for its weights, plus working memory for the conversation. If your memory can deliver, say, 100 GB per second, the upper limit is around 28 full passes through those weights per second, and therefore at most around 28 tokens per second, before any other overhead. Double the bandwidth and you roughly double that ceiling; double the model size and you halve it. TOPS alone does not make a fast chatbot.

The figure compares the three AI engines a typical laptop may offer, side by side, on the factors that matter for local AI: how much power each draws, what memory it reads from, how vendors rate it, which tasks suit it and how large a local model it can realistically hold. Read across each row to see where the trade-offs sit.

The pattern is clear: the NPU wins on efficiency, the integrated GPU on flexibility, and the discrete GPU on raw speed. For a local LLM, the deciding question is usually whether the model fits in fast memory, not which chip has the bigger TOPS number.

When the NPU is enough

The NPU is a good fit when the model is small, the task is steady rather than bursty, and the app has been built for it. Examples include operating system features, small language models tuned for NPUs, transcription and smart search over local files. There the NPU gives acceptable speed at very low power and leaves the GPU free.

06 · Software: How Apps Reach the NPU and GPU

Answer: Apps reach the NPU and GPU through runtimes such as ONNX Runtime, DirectML and Windows ML, vendor SDKs, and local LLM tools like Ollama and llama.cpp. The software path often decides which engine is used.

A model does not run on an NPU just because the NPU exists. On Windows, Microsoft provides common layers such as ONNX Runtime with hardware-specific execution providers, DirectML for GPU acceleration, and Windows ML as a higher-level path. Chip vendors add their own SDKs and tools to target their NPUs and GPUs. On Apple silicon, Core ML targets the Neural Engine, GPU and CPU.

Popular local LLM tools, such as Ollama and others built on llama.cpp, focus mainly on CPUs and GPUs. Their NPU support varies by vendor and is still evolving, so check current documentation for your chip and tool. A practical path to a local model:

1. Check your memory first. Note total RAM and, if you have one, discrete GPU VRAM. This sets the largest model you can realistically load.
2. Pick a model size that fits. Start with a small model in a quantized format (such as 4-bit), which needs far less memory than full precision.
3. Choose a runtime for your hardware. Use a local LLM tool that supports your GPU, or a vendor or ONNX Runtime path if you want the NPU.
4. Measure, do not assume. Record tokens per second, power draw and temperature on battery and on wall power.
5. Move up only if needed. If quality is too low, try a larger model; if it no longer fits or is too slow, consider the cloud.

07 · Power, Battery and Heat

Answer: The NPU uses the least energy for sustained AI work, protecting battery life. GPUs finish heavy jobs faster but draw much more power.

Battery life is where the NPU earns its place. Background blur may run for a two-hour meeting: on an NPU it is a small, steady load; on a GPU it keeps a larger engine busy, using more energy and making more heat and fan noise.

For bursty, heavy work the trade-off reverses: a GPU that finishes an image in seconds may use less total energy than a slower engine running much longer. Discrete laptop GPUs often slow down on battery, so a local LLM that feels fast at a desk can feel sluggish on a train.

08 · When to Use the Cloud Instead

Answer: Use cloud GPUs when the model does not fit in laptop memory, for training at scale, or when many users share one model. On OCI that means NVIDIA or AMD Instinct GPU instances.

Even a well-equipped laptop holds a fraction of the high-bandwidth memory (HBM) on data-center GPUs. Training, serious fine-tuning, serving a model to a whole team and the largest open models belong in the cloud. Oracle Cloud Infrastructure offers GPU instances with NVIDIA and AMD Instinct accelerators today, alongside CPU instances, and has announced Google TPUs from 2027, which our Google TPUs on OCI series covers in depth.

The decision flow below pulls the post together. Start at the top with any AI task and answer the questions in order: is it always-on, does it fit in memory, can the data leave the device, is battery the priority, and do you need fast replies from a bigger model? Each answer either settles where the task runs or moves you one step down.

Most everyday features end up on the NPU, most local LLM chat ends up on a GPU, and anything too large or too shared ends up in the cloud. Privacy is the main exception: if data cannot leave the device, shrink the model rather than moving it.

The NPU is how a laptop runs AI all day. The GPU is how it runs AI fast. The cloud is how you run AI that no laptop can hold.

Many teams end up hybrid: private features on the device, larger models on cloud GPUs. ExaGuru's hands-on OCI training covers the cloud half of that design.

09 · Key Takeaways

1. NPUs are for efficiency. They run small, always-on AI features such as blur and captions at very low power.
2. GPUs are for heavy AI. Local LLMs and image generation usually run best on a GPU, especially a discrete one.
3. TOPS is a rough guide. Vendor TOPS figures are peak, usually INT8, and not directly comparable across vendors.
4. Memory decides local LLMs. Bandwidth and capacity, not TOPS, set how big and how fast a local model can be.
5. The cloud covers the rest. Models too large for a laptop, training and shared serving belong on cloud GPUs such as OCI.

10 · Frequently Asked Questions