THE LAYOUT

A tour of a data-center GPU

A data-center GPU such as NVIDIA's H100 is a large die covered with repeated blocks called streaming multiprocessors (SMs). The H100 has up to 132 SMs enabled in its SXM version, sharing a large L2 cache and surrounded by stacks of HBM.

A GPU is one design block copied over a hundred times, plus the memory and links that feed it.

SIMT

Warps: 32 threads in lockstep

GPU programs launch thousands of threads. The hardware groups them into warps of 32 that run the same instruction together on different data. While one warp waits for memory, the SM switches to another, hiding the delay.

GPUs hide memory delays by always having other work ready.

TENSOR CORES

Matrix math in one instruction

🟦

Since the Volta generation (2017), NVIDIA GPUs include tensor cores: units that multiply small matrix tiles and add the result in a single operation. They deliver most of a modern GPU's AI performance, at FP16, BF16, FP8 and, with Blackwell, FP4.

Tensor cores turned the GPU from a parallel calculator into a matrix engine.

BY THE NUMBERS

H100 at a glance

Process TSMC 4N, about 80 billion transistors on roughly 814 mm²

Memory 80 GB of HBM3 at about 3.35 TB/s (SXM version)

Power Up to 700 W (SXM version)

Links NVLink at 900 GB/s to other GPUs

Its successor, Blackwell B200, joins two dies with 208 billion transistors and about 8 TB/s of HBM3e.

MANY GPUs AS ONE

Scaling out with NVLink

🔗

Big models do not fit on one GPU, so they are split across many. NVLink and NVSwitch chips connect GPUs within a server or rack so they can share work quickly; networks such as InfiniBand or Ethernet connect racks into clusters of tens of thousands of GPUs.

Modern AI is trained on whole buildings of GPUs acting as one computer.

UNIT 5 STUDY COMPLETE

🟩

Ready for the Fab Challenge?

You've walked through a data-center GPU from its streaming multiprocessors and tensor cores out to its HBM and NVLink connections.