THE BUILDING BLOCK
One artificial neuron
An artificial neuron multiplies each input by a weight, adds the results together, adds a bias, and passes the total through a simple function such as ReLU (keep it if positive, otherwise zero).
THE KEY OPERATION
The multiply-accumulate
Multiply
An input times its weight, such as 0.8 × 0.5
Accumulate
Add the product to a running total kept in a register
Repeat
Once for every input to every neuron in every layer
LAYERS AS MATRICES
A whole layer is a matrix multiply
Stack the weights of every neuron in a layer into a grid, a matrix. Then a whole layer becomes one matrix multiplication: the input vector times the weight matrix. Large language models are mostly giant matrix multiplications, one after another.
HOW MANY?
Counting the MACs
A layer with 4,096 inputs and 4,096 outputs needs 4,096 × 4,096 ≈ 16.8 million MACs for every token it processes. A model with billions of weights does roughly one MAC per weight per token.
TRAINING VS. INFERENCE
Learning vs. answering
Training
Show the model examples, measure the error, and nudge every weight. Roughly three times the math of a forward pass, repeated over trillions of words
Inference
Run the finished model forward to answer a question. Less math per step, but done billions of times a day
Different needs
Training wants huge clusters and fast links; inference wants low cost and low delay per answer
UNIT 1 STUDY COMPLETE
Ready for the Fab Challenge?
You've seen that a neural network is layers of weighted sums, and that each one is built from multiply-accumulate operations.