Mid-level interview question
Explain absmax INT8 quantization. What's the worst-case rounding error, and what does a single outlier do?
Model answer
Divide each value by the group's largest magnitude, multiply by 127, round; store the integers plus one scale. A value moves at most half a step.
Scroll sideways to see the whole diagram
Start with real weights: the first 12 of the 768 numbers in GPT-2's embedding row for the token " bank".
Text version
- Start with real weights: the first 12 of the 768 numbers in GPT-2's embedding row for the token " bank".
- Find the largest magnitude in the group. Here it is 0.344.
- Map it to the largest INT8 value, 127. The scale is the size of one integer step: 0.344 / 127, about 0.00271.
- Divide every value by the scale and round to the nearest integer. 0.137 becomes 51.
- Store the integers, one byte each, plus the one scale for the whole group.
- To use a weight, multiply back: 51 × 0.00271 is 0.1381, not 0.137. That 0.0011 is the quantization error.
- Rounding moves each value at most half a step, 0.00135 here. A bigger largest value means a bigger step and a bigger error for every value in the group.
Real values: 12 numbers from GPT-2's embedding row for " bank" (lib/labs/data/running-example.json, stored to 3 decimals). Method: absmax quantization, LLM.int8() (Dettmers et al., 2022), §2.1.
One outlier widens the step for its whole group: adding an invented value 20 times our largest GPT-2 weight took the step from 0.00271 to 0.0542, and the largest error on the real weights from 0.00126 to 0.027. Hence per-block scales.
Practise the rest
7 Quantization questions, 492 in all, each with a model answer. Free.
More Quantization interview questions
- How much memory do a model's weights take, and what does quantizing GPT-2 small to INT4 save?
- FP16 and BF16 both use 16 bits. How do they differ, and how can loading a model in the wrong one hurt?
- You must serve an open 13B model on one 80 GB accelerator and want more concurrent conversations. Which quantization would you use, and what does it actually buy?
- A colleague says INT4 weights will make decode eight times faster because they're eight times smaller. How do you respond?
- Implement FP32-to-int8 quantization in NumPy with a scale, zero point and clipping, and the matching dequantizer.
- Explain quantization-aware training, knowledge distillation, eval mode and contrastive learning. How would you verify that each one improved a production model?