Mid-level interview question
Why does full fine-tuning need so much memory, and what does LoRA change?
Model answer
Besides the weights, you store gradients and optimizer states for every trained parameter: a 7B model in 16-bit with Adam is 14 GB plus 42 GB. LoRA trains small low-rank matrices instead, merged back without inference latency, so training memory shrinks.
Scroll sideways to see the whole diagram
The pretrained weight matrix W stays frozen: no gradients, no optimizer state for it.
Text version
- The pretrained weight matrix W stays frozen: no gradients, no optimizer state for it.
- LoRA adds a thin side path. A squeezes the input down to r numbers and B expands it back; only A and B are trained.
- After training, B·A can be added into W, so the fine-tuned model runs exactly as fast as the original.
Hu et al. (2021), LoRA: Low-Rank Adaptation of Large Language Models. Counts are for one 4,096 × 4,096 matrix at rank 8.
Practise the rest
6 When to fine-tune questions, 492 in all, each with a model answer. Free.