Quantization
Also called quantized, quantize
Storing a model's numbers in a format with fewer bits, such as FP16, INT8 or INT4, to cut memory and the bytes each step reads. Strictly it means converting to integers, but in practice it covers any precision reduction.
Interview questions about Quantization
- Explain absmax INT8 quantization. What's the worst-case rounding error, and what does a single outlier do?
Divide each value by the group's largest magnitude, multiply by 127, round; store the integers plus one scale. A value moves at most half a step.
- You must serve an open 13B model on one 80 GB accelerator and want more concurrent conversations. Which quantization would you use, and what does it actually buy?
Post-training, weight-only quantization. I didn't train the model, PTQ is the common path frameworks offer in a few lines, and weight quantization affects…
- Implement FP32-to-int8 quantization in NumPy with a scale, zero point and clipping, and the matching dequantizer.
- Explain quantization-aware training, knowledge distillation, eval mode and contrastive learning. How would you verify that each one improved a production model?
Quantization-aware training simulates low-precision rounding during fine-tuning, so accuracy holds better than post-training quantization.
- Cover quantization, pruning, distillation and kernel fusion, then compare tensor and pipeline parallelism.
Quantization stores weights or activations in fewer bits. Pruning removes weights, and only structured sparsity such as 2:4 speeds up GPUs.
- Design and implement a KV cache for autoregressive decoding in a Transformer: tensor shapes for prefill and per-step decode, per-layer caches, batching, memory limits and beam search.
- Your AI chatbot takes too long to respond. What do you do?
First measure which wait it is: time to first token or total time, at p50 and p99. Slow first token: shorten the prompt, cache the static prefix, cut queueing.
- Design optimised inference pipelines, and tell me how you would benchmark them.
Benchmark the workload you'll serve: realistic prompt and output length distributions, arrival rates and concurrency, not one fixed prompt.