Senior interview question
You must serve an open 13B model on one 80 GB accelerator and want more concurrent conversations. Which quantization would you use, and what does it actually buy?
Model answer
Post-training, weight-only quantization. I didn't train the model, PTQ is the common path frameworks offer in a few lines, and weight quantization affects quality more stably.
It buys less than the byte ratio suggests. For Llama 2 13B at 2,048 tokens with a 16-bit cache, INT4 weights take 6.5 GB instead of 26, so 43 conversations fit instead of 32: the cache takes most of the memory. Next lever: shrink the cache. Hands-On LLMs' rule of thumb: stay at 4 bits or above.
Practise the rest
7 Quantization questions, 492 in all, each with a model answer. Free.
More Quantization interview questions
- How much memory do a model's weights take, and what does quantizing GPT-2 small to INT4 save?
- FP16 and BF16 both use 16 bits. How do they differ, and how can loading a model in the wrong one hurt?
- Explain absmax INT8 quantization. What's the worst-case rounding error, and what does a single outlier do?
- A colleague says INT4 weights will make decode eight times faster because they're eight times smaller. How do you respond?
- Implement FP32-to-int8 quantization in NumPy with a scale, zero point and clipping, and the matching dequantizer.
- Explain quantization-aware training, knowledge distillation, eval mode and contrastive learning. How would you verify that each one improved a production model?