THE SURPRISE

Math is cheap, moving data is not

In a classic 45 nm comparison by Mark Horowitz, a 32-bit floating-point multiply cost about 4 pJ, while reading 32 bits from off-chip DRAM cost around 640 pJ. That is more than a hundred times as much.

Most of an AI chip's energy can go into moving numbers, not multiplying them.

THE MEMORY HIERARCHY

Near is fast, far is slow

Registers Right beside the math units; almost free to read

SRAM (on chip) Tens to hundreds of megabytes; fast but takes a lot of area

HBM (beside the chip) Tens to hundreds of gigabytes; several terabytes per second

DRAM / SSD (far away) Huge but slow and costly in energy per bit

AI chip design is a game of keeping the data that is needed next as close to the math as possible.

THE WALL

Compute grew faster than memory

📈

Over the past two decades, the peak math speed of chips has grown much faster than the bandwidth of the memory that feeds them. The gap is called the memory wall. Large language models feel it most when answering, because every weight must be read for every new token.

For many AI jobs the limit is not how fast a chip can multiply, but how fast it can be fed.

THE ROOFLINE

Compute-bound or memory-bound?

Arithmetic intensity is the number of operations done for each byte fetched. Below a certain intensity a chip waits on memory; above it, the math units are the limit. This picture is called the roofline model.

Reuse every number you fetch as many times as you can, and you climb the roof.

BREAKING THE WALL

Tricks to feed the beast

🛠️

♻️

Reuse

Keep a weight on chip and use it for many inputs (batching)

🥞

Stack memory close

HBM sits millimetres from the chip on the same package

🔽

Fewer bits

Smaller numbers mean fewer bytes to move

Every major AI chip idea in this course is, at heart, a way around the memory wall.

UNIT 3 STUDY COMPLETE

🧱

Ready for the Fab Challenge?

You've seen that fetching data from DRAM costs over a hundred times more energy than a multiply, and how arithmetic intensity decides whether a chip is starved for data.