Lesson 3 · 7 topics · 20 min · intermediate

Precision and Quantization: Fewer Bytes per Weight

You will be able to

Compute a model's weight memory from its parameter count and bits per weight.
Explain how FP32, FP16 and BF16 split their bits between range and precision, and what goes wrong in each.
Quantize numbers to INT8 or INT4 with an absmax scale and bound the rounding error.
Explain why one outlier hurts a whole group, and why schemes use many scales.
Tell weight-only from weight-and-activation quantization, and post-training quantization from quantization-aware training.
Estimate how many conversations fit with quantized weights, and what decode speed-up fewer bytes could give, labelled as an estimate.

Topics

Your review

Sign in to keep your quick-check answers. The ones you miss come back for review after 1, 3 and 7 days, until they stick.

Sources

AI Engineering · Chip Huyen, 2025
Chip Huyen. O'Reilly. Used: Ch. 7 "Finetuning", "Memory Bottlenecks": "Numerical Representations", "Quantization" (sidebar "Quantization versus Reduced Precision"), "Inference quantization", "Training quantization"; Ch. 9 "Inference Optimization", "Model Optimization": "Model compression".
Hands-On Large Language Models · Jay Alammar et al., 2024
Jay Alammar, Maarten Grootendorst. O'Reilly. Used: Ch. 7 "Advanced Text Generation Techniques and Tools": "Model I/O: Loading Quantized Models with LangChain"; Ch. 12 "Fine-Tuning Generation Models": "Compressing the model for (more) efficient training".
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale · Tim Dettmers et al., 2022
Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer. NeurIPS 2022. Used: §1 Introduction; §2 Background, §2.1 "8-bit Data Types and Quantization"; §3 "Int8 Matrix Multiplication at Scale". Open source
Train With Mixed Precision · NVIDIA
NVIDIA. NVIDIA Deep Learning Performance documentation. Used: §2.1 "Half Precision Format". Open source
BFloat16: The secret to high performance on Cloud TPUs · Shibo Wang et al., 2019
Shibo Wang, Pankaj Kanwar. Google Cloud Blog. Used: Bit layout of bfloat16; same exponent size as FP32. Open source
Improve your model's performance with bfloat16 · Google Cloud
Google Cloud. Cloud TPU documentation. Used: "The dynamic range of bfloat16 and float32 are equivalent.". Open source

You will be able to

Compute a model's weight memory from its parameter count and bits per weight.

Explain how FP32, FP16 and BF16 split their bits between range and precision, and what goes wrong in each.

Quantize numbers to INT8 or INT4 with an absmax scale and bound the rounding error.

Explain why one outlier hurts a whole group, and why schemes use many scales.

Tell weight-only from weight-and-activation quantization, and post-training quantization from quantization-aware training.

Estimate how many conversations fit with quantized weights, and what decode speed-up fewer bytes could give, labelled as an estimate.

Sources

AI Engineering · Chip Huyen, 2025

Chip Huyen. O'Reilly. Used: Ch. 7 "Finetuning", "Memory Bottlenecks": "Numerical Representations", "Quantization" (sidebar "Quantization versus Reduced Precision"), "Inference quantization", "Training quantization"; Ch. 9 "Inference Optimization", "Model Optimization": "Model compression".

Hands-On Large Language Models · Jay Alammar et al., 2024

Jay Alammar, Maarten Grootendorst. O'Reilly. Used: Ch. 7 "Advanced Text Generation Techniques and Tools": "Model I/O: Loading Quantized Models with LangChain"; Ch. 12 "Fine-Tuning Generation Models": "Compressing the model for (more) efficient training".

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale · Tim Dettmers et al., 2022

Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer. NeurIPS 2022. Used: §1 Introduction; §2 Background, §2.1 "8-bit Data Types and Quantization"; §3 "Int8 Matrix Multiplication at Scale". Open source

Train With Mixed Precision · NVIDIA

NVIDIA. NVIDIA Deep Learning Performance documentation. Used: §2.1 "Half Precision Format". Open source

BFloat16: The secret to high performance on Cloud TPUs · Shibo Wang et al., 2019

Shibo Wang, Pankaj Kanwar. Google Cloud Blog. Used: Bit layout of bfloat16; same exponent size as FP32. Open source

Improve your model's performance with bfloat16 · Google Cloud

Google Cloud. Cloud TPU documentation. Used: "The dynamic range of bfloat16 and float32 are equivalent.". Open source