Topic 5
Backpropagation: a gradient for every weight
Gradient descent needs a gradient for every parameter, on every step. Backpropagation is how a network gets them all at once, and AI Engineering splits each training step into two phases.
- Forward pass: compute the output from the input, layer by layer.
- Backward pass: compare the output with the expected one (the loss), then work out how much each parameter contributed to the mistake. That contribution is the parameter's gradient, the derivative of the loss with respect to it, and there is one per parameter. A parameter with a large gradient contributed a lot and gets adjusted more.
The backward pass works from the output towards the inputs. The output neuron's error is known directly; each earlier neuron's share of the blame is the error of the neurons it feeds, weighted by the connections between them. A weight's gradient is then its neuron's share of the blame times the input that weight saw. One pass backwards, and every weight has its gradient.
Scroll sideways to see the whole diagram
AI Engineering: each training step has a forward pass, computing the output from the input, and a backward pass. Inference runs only the forward pass.
Text version
- AI Engineering: each training step has a forward pass, computing the output from the input, and a backward pass. Inference runs only the forward pass.
- Compare the outputs with the expected outputs. The difference is the loss.
- Work out how much each trainable parameter contributed to the mistake: its gradient, the derivative of the loss with respect to that parameter. There is one gradient per parameter.
- The optimizer adjusts each parameter using its gradient. SGD subtracts learning rate × gradient; Adam, the usual choice for transformers, also keeps running averages per parameter.
- Repeat with the next batch. The loss curve over many steps is how you watch training.
What happens to the gradients is the optimizer's job. Plain SGD (stochastic gradient descent) subtracts learning rate × gradient, as in the last two topics. The book notes that Adam is by far the most widely used optimizer for transformer models; it also keeps running averages for each parameter, which is why training needs more memory than inference.
This lesson's network is written from scratch, and its tests check the backward pass the slow way: nudge one weight by a millionth, measure how much the loss changes, and compare that with the gradient backpropagation computed. They agree.
Quick check