Junior interview question
FP16 and BF16 both use 16 bits. How do they differ, and how can loading a model in the wrong one hurt?
Model answer
They split the bits differently: FP16 has 5 exponent and 10 fraction bits, BF16 8 and 7. Exponent bits buy range, fraction bits precision, so FP16 is finer but tops out at 65,504, while BF16 keeps FP32's range. Llama 2 shipped in BF16; teams loading it in FP16 found quality much worse than advertised.
Practise the rest
7 Quantization questions, 492 in all, each with a model answer. Free.
More Quantization interview questions
- How much memory do a model's weights take, and what does quantizing GPT-2 small to INT4 save?
- Explain absmax INT8 quantization. What's the worst-case rounding error, and what does a single outlier do?
- You must serve an open 13B model on one 80 GB accelerator and want more concurrent conversations. Which quantization would you use, and what does it actually buy?
- A colleague says INT4 weights will make decode eight times faster because they're eight times smaller. How do you respond?
- Implement FP32-to-int8 quantization in NumPy with a scale, zero point and clipping, and the matching dequantizer.
- Explain quantization-aware training, knowledge distillation, eval mode and contrastive learning. How would you verify that each one improved a production model?