FLOATING POINT

Sign, exponent, mantissa

A floating-point number is stored in three parts: a sign bit, an exponent that sets its size, and a mantissa that holds its digits. More exponent bits give more range; more mantissa bits give more precision.

Choose the bits to fit the job: range and precision are traded against size.

WHY FEWER BITS

Halve the bits, double the work

⬇️

📦

Less memory

A 70-billion-weight model needs 280 GB at 32 bits but 35 GB at 4 bits

🚚

Less traffic

Fewer bytes per weight eases the memory wall

🔌

Smaller multipliers

A multiplier's area and energy shrink much faster than its bit width

Lower precision is the cheapest speed-up in AI hardware.

THE FORMATS

A ladder of number formats

🪜

FP32 1 sign, 8 exponent, 23 mantissa: the classic scientific format

FP16 1, 5, 10: half the size, but a narrow range

BF16 1, 8, 7: Google's 'brain float' keeps FP32's range with fewer digits

FP8 E4M3 or E5M2: added for training and inference with NVIDIA's Hopper (2022)

FP4 E2M1, with shared scale factors: supported by NVIDIA's Blackwell (2024)

Each step down the ladder roughly doubled the peak math a chip could do.

INTEGERS TOO

INT8 and quantization

🎚️

Quantization converts a trained model's weights to small integers such as INT8 or INT4, with a scale factor per group of weights. Done carefully, accuracy barely changes, and the model runs on far cheaper hardware, even phones.

Training usually needs more precision than inference, so models are often trained at 16 or 8 bits and served at 8 or 4.

THE CATCH

How low can you go?

Too few bits and numbers overflow, underflow or round away small but important differences. Chips fight this with mixed precision: multiply in a small format but add up results in a bigger one, and keep a scale factor for each small block of numbers.

Multiply small, accumulate big: that is how a 4-bit chip still gets the right answer.

UNIT 4 STUDY COMPLETE

🔢

Ready for the Fab Challenge?

You've seen how a floating-point number is split into sign, exponent and mantissa, and why halving the bits roughly doubles the math per watt.