Topic 2

Loss: one number for how wrong

To improve a model you first need to measure how wrong it is, as a single number. That number is the loss. AI Engineering defines it as the difference between the computed output and the expected output: if they differ, the model made a mistake and its parameters need adjusting.

The network in this lesson outputs a number between −1 and 1, and each training point's right answer is +1 or −1. Its loss for one point is half the squared distance between the two, ½(output − answer)², and the loss over a dataset is the average across its points.

Loss: how far off, squaredStep 1 / 4

Scroll sideways to see the whole diagram

This lesson's network outputs a number between −1 and 1; the right answer for each point is +1 or −1.

Text version

This lesson's network outputs a number between −1 and 1; the right answer for each point is +1 or −1.
Close to the right answer: a small loss.
Squaring makes big misses cost far more: 0.4 off costs ½ × 0.16 = 0.08, but 1.6 off costs ½ × 2.56 = 1.28. Four times the distance, sixteen times the loss.
Average over all the points and you have one number to make smaller. Training is the search for parameters that make it small.

Squaring does two things. Every miss counts as positive, whichever side it falls on, and big misses cost far more than small ones, so training works hardest on the points the model gets most wrong.

Quick check

For a point whose answer is −1, which output gives the larger loss: 0.5 or −0.2?

−0.2, because it's further from zeroThey're equal, because both are less than 1 away from zero0.5, because it's 1.5 away from −1; −0.2 is only 0.8 awayNeither: loss is only counted when the sign is wrong

Language models use a different loss, cross entropy, because their output is a probability for every token rather than one number (the last topic shows it). The idea is the same: one number that gets smaller as the model gets better, and training's job is to push it down.