Junior interview question
How much memory do a model's weights take, and what does quantizing GPT-2 small to INT4 save?
Model answer
Parameters × bits per weight ÷ 8. GPT-2 small's 124,439,808 weights take 497.8 MB in FP32 and 62.2 MB in INT4, an eighth, plus small per-group scales. Decode reads every weight each step, so bytes per weight set both the memory and the bytes each step moves.
Practise the rest
7 Quantization questions, 492 in all, each with a model answer. Free.
More Quantization interview questions
- FP16 and BF16 both use 16 bits. How do they differ, and how can loading a model in the wrong one hurt?
- Explain absmax INT8 quantization. What's the worst-case rounding error, and what does a single outlier do?
- You must serve an open 13B model on one 80 GB accelerator and want more concurrent conversations. Which quantization would you use, and what does it actually buy?
- A colleague says INT4 weights will make decode eight times faster because they're eight times smaller. How do you respond?
- Implement FP32-to-int8 quantization in NumPy with a scale, zero point and clipping, and the matching dequantizer.
- Explain quantization-aware training, knowledge distillation, eval mode and contrastive learning. How would you verify that each one improved a production model?