AI IN YOUR POCKET

The neural processing unit

📱

A neural processing unit (NPU) is a small AI accelerator inside a larger chip. Apple added its Neural Engine to the A11 chip in the iPhone 8 and iPhone X (2017); Qualcomm, Google, Samsung, MediaTek, Intel and AMD now build NPUs into phone and laptop chips.

The NPU sits beside the CPU and GPU on the same die, sharing the same memory.

WHY ON DEVICE?

Reasons to stay local

🔒

Privacy

Photos, voice and messages never leave the device

⚡

Speed

No round trip to a data center

📴

Offline

Works on a plane or in a tunnel

Face unlock, photo processing, live captions and small language models all run on NPUs.

THE BUDGET

Watts matter more than speed

A phone chip may have a few watts in total, shared by everything. NPUs therefore use low precision (INT8, INT4), lots of local SRAM and aggressive power gating. Their speed is often quoted in TOPS: trillions of operations per second. Microsoft's Copilot+ PC label (2024) requires an NPU of 40 TOPS or more.

At the edge, TOPS per watt is the number that matters.

WHERE ELSE

Edge AI everywhere

Cars Driver assistance chips process camera and radar data in real time

Cameras Smart cameras spot people and objects without streaming video

Wearables Watches and earbuds detect heart rhythm or keywords on microwatts to milliwatts

Robots and drones Navigation and vision on a battery

The smallest AI chips may run on a coin cell for years.

EDGE + CLOUD

Splitting the work

☁️

Many products split the job: a small model on the device handles quick or private tasks, and harder requests go to a big model in the cloud. The same tricks, such as quantization and distillation into smaller models, make AI fit on both.

Cloud chips chase raw scale; edge chips chase efficiency.

UNIT 8 STUDY COMPLETE

📱

Ready for the Fab Challenge?

You've seen why AI runs on devices as well as in the cloud, and how small NPUs squeeze it into a battery budget.