Topic 4

The learning rate

The learning rate is the setting people tune most. AI Engineering calls it the step size: too small and training takes too long to reach the goal, too big and it can overstep the goal and never converge.

On the curve loss = w², each step multiplies w by (1 − 2 × learning rate), which makes the effect exact and easy to see.

Predict first

Starting from w = 2.5, what does a learning rate of 1.05 do to w over 20 steps?

It reaches the minimum faster than any smaller rateIt settles slightly to one side of the minimumIt bounces between 2.5 and −2.5 foreverIt moves further from the minimum on every step

Same start, same 20 steps, different learning rates

Illustrative · loss = w² · start at w = 2.5 · 20 steps

After 20 steps, w

0.03

Loss

0.0008

Settles at the minimum

On this curve one step multiplies w by (1 − 2 × learning rate). Below 1, the steps shrink; at exactly 1, w flips sign forever; above 1, it grows.

There is no universal best value. The book suggests experimenting, and reading the loss curve: if it jumps around a lot, the learning rate is probably too big; if it's steady but slow to fall, probably too small. Many training runs also change the rate as they go, larger at the start and smaller near the end, using a learning rate schedule.

The size of the step isn't the only thing that decides where you end up. Gradient descent only knows the slope where it stands, so on a landscape with more than one valley, the starting point decides which valley it settles in.

Two valleys: which one do you reach?

Illustrative curve · learning rate 0.05 · 80 steps

Starting weight: 1.0

Ends at w

2.07

Final loss · shallower valley

1.79

Gradient descent only sees the slope where it stands, so it settles in the nearest valley, not necessarily the deepest.

Quick check

A loss curve falls smoothly but very slowly over thousands of steps. What does that suggest?

The learning rate is probably too smallThe learning rate is probably too bigThe model has reached its best possible lossThe data has no pattern to learn