Topic 3

Gradient descent: downhill in small steps

Picture the loss as a landscape: every possible setting of the parameters is a place, and the loss is the height there. Training wants the lowest point. With millions of parameters you can't look at the whole landscape, but you can always feel the slope under your feet.

That slope is the gradient: how much the loss changes when a parameter changes a little. Gradient descent takes a small step downhill, against the gradient, then measures the slope again. For one weight:

new weight = weight − learning rate × gradient

The learning rate sets the step size. Start with the simplest landscape there is, loss = w², whose slope at any point is 2w.

Predict first

At w = 2.5 the slope of loss = w² is 2 × 2.5 = 5. With a learning rate of 0.1, where is w after one step?

2.42.00.5−2.5

Walk downhill, one step at a time

Illustrative · loss = w² · start at w = 2.5 · learning rate 0.1

w ← w − learning rate × slope = 2.50 − 0.1 × 5.00 = 2.00

Steps

Weight w

2.50

Loss

6.25

The slope (gradient) says which way is uphill and how steep it is. Each step moves the weight a little the other way.

A real network does exactly this, for every parameter at once. Its landscape has one dimension per parameter, so you can't draw it, but each weight still moves a little against its own gradient on every step.