Topic 7

Epochs, batches and overfitting

Two more training settings matter as much as the learning rate. AI Engineering defines them:

• Batch size: how many examples each update learns from. Very small batches (the book says fewer than eight) can make training unstable; larger ones give steadier updates but need more memory.
• Epochs: an epoch is one pass over the training data, so the number of epochs is how many times each example is trained on.

More epochs aren't always better. A model can start memorising its training examples, including their noise, instead of learning the pattern. That's overfitting, and the only way to see it is to keep some data back and measure the loss on that too. The book's rule: if training loss still falls while validation loss rises, the model is overfitting, and you might try fewer epochs.

Three fixes, each visible in the lab:

• Stop earlier. The model at epoch 16 was the best one this run produced. Keeping the checkpoint with the lowest validation loss is called early stopping.
• Use more data. With 150 training points instead of 30, the same big network's lowest test loss is 0.185.
• Use a smaller model. With 9 parameters there isn't room to memorise 30 points: final test loss 0.241 and 83.7% test accuracy, against 0.547 and 68.5% for the big network.

The playground also has regularization (L1 and L2), which pulls every weight towards zero a little on each step. On this example it helps less than the three fixes above, so try it there and compare.