What is a phone's neural engine, and what does it do?

What the hardware actually does

Strip a language model down and you find nearly all of it is one operation: multiply two numbers, add the result to a running total, and repeat that billions of times. Together those steps make a matrix multiplication. A neural accelerator may include specialized units for matrix and multiply-accumulate operations. The exact design varies by chip and is not fully described by its product name.

Specialized hardware can reduce overhead for supported operations, while CPUs and GPUs offer different flexibility and performance tradeoffs. Actual energy use depends on the chip, model, and runtime.

The names differ by vendor. Neural engine, NPU, AI accelerator, tensor block: roughly the same idea, a narrow block on the same die as the CPU and GPU in your phone, sharing its memory with them.

What it buys is energy per operation

Moving data can consume more energy than arithmetic in many workloads, which is one reason designers use local caches and specialized accelerators. The exact balance varies with the chip and process; a single old measurement should not be treated as a current phone benchmark.

Mobile devices have finite battery and thermal budgets. Sustained workloads can trigger performance limits; the effect depends on the device, workload and operating conditions.

CPU, GPU, or neural engine?

All three can run a neural network. They differ in supported operations, flexibility, throughput, and energy use, with results depending on the chip and workload. Flexibility and efficiency trade against each other in silicon, and the three blocks sit at different points along that trade.

Which one is best depends on the shape of the work and on what the runtime supports. The choice is rarely permanent. A single request can start on the CPU, run its heavy layers on the accelerator, and fall back to the GPU for a step the accelerator does not implement. All three read from the same pool of memory.

• CPU: flexible for general and irregular work; performance and energy use depend on the chip and workload.
• GPU: supports parallel workloads and offers different flexibility and performance tradeoffs from a CPU or accelerator.
• Neural accelerator: may speed supported operations efficiently, but coverage and results depend on its design, precision support and runtime.

Why a phone with an accelerator can still feel slow

Generating text has two phases with different bottlenecks. Reading your prompt is dense parallel math, and the accelerator handles it well. Producing the reply is not: each token needs a fresh trip out to memory for its weights, so the accelerator's own multipliers finish their work and sit idle waiting for the next one to arrive. The pace is set by memory bandwidth, not by how many operations per second the hardware can do.

The second limit is coverage, and this one belongs to the accelerator alone. The set of operations it can perform is frozen in silicon when the chip is designed, years before any given model exists, so it can only ever run what its designers anticipated. A model built around an operation the block does not implement falls back to the CPU or GPU for that piece, and a fallback in the middle of the network can cost more in the round trip between processors than the accelerator saved everywhere else. A chip only a few years old can struggle with a design newer than itself, because the model changed after the silicon shipped, not the other way around. Models get shaped to fit the hardware as much as the reverse.

Does dedicated AI hardware mean a phone can run any model?

No, and the accelerator is not the reason. It has no meaningful memory of its own beyond small working buffers. Every weight it multiplies is read from the same RAM the operating system and your other apps are using. What fits on your phone is a memory question before it is a compute question.

Accelerators have been shipping in phone chips for years, and speech has long been their most common real workload: wake-word detection and dictation ran on this same kind of block, at a far smaller scale, before a language model ever did. What changed for language models is the software side: models small enough, and quantized hard enough, to fit in the memory a phone can spare while staying worth using. The hardware was there first.

The constraints described on this page apply to any on-device workload: a processor shares memory and energy with everything else the device is doing. The balance depends on the hardware and task.