Senior interview question
A colleague says INT4 weights will make decode eight times faster because they're eight times smaller. How do you respond?
Model answer
It's a ceiling, not a forecast. Batch-1 decode is bandwidth-bound, so time should track bytes read, but the KV cache doesn't shrink. Scaling our measured 9.1 ms CPU step by bytes gives about 1.3 ms at INT4: an estimate, not a measurement, with the cache now 13% of the bytes.
And the format needs hardware and kernel support: format conversion costs compute, and LLM.int8() claims only modest matrix-multiply speedups. I'd measure step time on our setup.
Practise the rest
7 Quantization questions, 492 in all, each with a model answer. Free.
More Quantization interview questions
- How much memory do a model's weights take, and what does quantizing GPT-2 small to INT4 save?
- FP16 and BF16 both use 16 bits. How do they differ, and how can loading a model in the wrong one hurt?
- Explain absmax INT8 quantization. What's the worst-case rounding error, and what does a single outlier do?
- You must serve an open 13B model on one 80 GB accelerator and want more concurrent conversations. Which quantization would you use, and what does it actually buy?
- Implement FP32-to-int8 quantization in NumPy with a scale, zero point and clipping, and the matching dequantizer.
- Explain quantization-aware training, knowledge distillation, eval mode and contrastive learning. How would you verify that each one improved a production model?