Topic 5
What fine-tuning takes: memory and LoRA
If the failures say fine-tune, the first practical question is memory. AI Engineering (Ch. 7, "Parameter-Efficient Finetuning") works an example: a 7-billion-parameter model in a 16-bit format needs 14 GB just for its weights. Full fine-tuning with the Adam optimizer, also in 16-bit, needs another 7B × 3 × 2 bytes = 42 GB for gradients and optimizer states. That's 56 GB before activations, more than most consumer GPUs, which the book notes have 12–24 GB.
Predict first
Will it fit on one GPU?
AI Engineering Ch. 7's arithmetic · weights + gradients + Adam's two states · activations not included
Weights
14.0 GB
Gradients + optimizer
42.00 GB
Total (before activations)
56.0 GB
Fits a 24 GB GPU?
no
Drag the trainable share down. At 1%, the gradients and optimizer states shrink from 42 GB to 0.42 GB; the 14 GB of frozen weights remain.
LoRA (Hu et al., 2021), which the book covers in detail, is the best-known way to train a tiny share. For a weight matrix, it trains two small matrices whose product is added to the frozen original, and those can be merged back afterwards, so unlike some adapter methods it adds no inference latency.
Scroll sideways to see the whole diagram
The pretrained weight matrix W stays frozen: no gradients, no optimizer state for it.
Text version
- The pretrained weight matrix W stays frozen: no gradients, no optimizer state for it.
- LoRA adds a thin side path. A squeezes the input down to r numbers and B expands it back; only A and B are trained.
- After training, B·A can be added into W, so the fine-tuned model runs exactly as fast as the original.
Hu et al. (2021), LoRA: Low-Rank Adaptation of Large Language Models. Counts are for one 4,096 × 4,096 matrix at rank 8.
None of this removes the larger costs from the last page: the data, the evaluation, and the maintenance. LoRA makes fine-tuning cheaper to run, not cheaper to get right.
Quick check