How a Network Learns: Loss and Gradient Descent

You will be able to

• Describe a neural network as a function whose behaviour is set by its parameters (weights and biases), and count a small network's parameters.
• Define loss as one number for how wrong a model is, and compute squared error for a prediction.
• Apply gradient descent by hand: move a weight against its gradient, scaled by the learning rate.
• Predict what too small, well chosen and too large learning rates do to training, and read the loss curve for hints.
• Explain the forward and backward pass of backpropagation and what an optimizer such as SGD or Adam does with the gradients.
• Explain why hidden layers and nonlinear activations let a network learn shapes a straight line cannot, such as XOR.
• Recognise overfitting from training and test loss, and name three fixes: stop earlier, use more data, use a smaller model.

Topics

1. 1A model is a function with knobs
2. 2Loss: one number for how wrong
3. 3Gradient descent: downhill in small steps
4. 4The learning rate
5. 5Backpropagation: a gradient for every weight
6. 6Why hidden layers: XOR
7. 7Epochs, batches and overfitting
8. 8From this network to an LLM