Topic 8
From this network to an LLM
Everything in this lesson scales up to language models. The differences are in the data, the size and the loss, not in the loop.
Scroll sideways to see the whole diagram
AI Engineering: language models train by self-supervision. Each sequence supplies its own labels, the tokens to predict, so nobody has to label the data by hand.
Text version
- AI Engineering: language models train by self-supervision. Each sequence supplies its own labels, the tokens to predict, so nobody has to label the data by hand.
- The book's example of scale: GPT (June 2018) had 117 million parameters; GPT-2 (February 2019), 1.5 billion.
- A language model's output is a probability for every token in the vocabulary, so its loss is cross entropy: how surprised it was by the real next token.
- But each training step is still a forward pass, a loss, a backward pass and an update, the loop you can watch in the playground.
GPT and GPT-2 sizes as reported in AI Engineering (Huyen), Ch. 1 "Self-supervision".
The biggest difference is where the labels come from. Here every point was labelled +1 or −1 in advance. AI Engineering explains that language models train by self-supervision: each sequence of text supplies its own labels, the next token at every position, so there's no labelling bottleneck and the training data can be enormous. A model trained that way is trained to minimise its cross entropy on that data.
Quick check
Try it yourself
The playground has everything from this lesson in one place: four datasets, seven inputs, up to six hidden layers, four activation functions, the learning rate, batch size, noise and regularization. Some things to try:
- Spiral with only x₁ and x₂: how many layers does it take?
- Any dataset at a learning rate of 3 or 10: watch the loss curve.
- ReLU against tanh on the circle.
Summary
- A model is a function whose behaviour is set by parameters: weights and biases.
- Loss is one number for how wrong the model is; training pushes it down.
- Gradient descent moves each parameter a little against its gradient, scaled by the learning rate. Too small is slow; too big overshoots or blows up.
- Backpropagation computes every parameter's gradient in one backward pass; an optimizer such as SGD or Adam applies them. Inference runs only the forward pass.
- Hidden layers with nonlinear activations, or better inputs, let a network learn shapes a straight line can't.
- Watch held-back data: training loss falling while test loss rises means overfitting. Stop earlier, add data or use a smaller model.
Next
Next lesson: tokenization, the first step of the forward pass for a language model: turning text into the numbers a network like this one can read.