BF16 (bfloat16)
Also called bf16, bfloat16
A 16-bit float with 1 sign, 8 exponent and 7 fraction bits. It has the same range as FP32 but less precision than FP16.
Interview questions about BF16 (bfloat16)
- FP16 and BF16 both use 16 bits. How do they differ, and how can loading a model in the wrong one hurt?
They split the bits differently: FP16 has 5 exponent and 10 fraction bits, BF16 8 and 7.
- Walk me through Transformer attention: causal and padding masks, cross-attention, why a KV cache helps at inference, and how mixed-precision training works.
Masks set scores to minus infinity before the softmax. The causal mask hides future positions; the padding mask hides pad keys, and pads are excluded from the…
- How would you optimise large-scale training and inference on GPUs?
Profile first, then attack the binding limit. Training: BF16 or FP8 mixed precision, FlashAttention and fused kernels, activation checkpointing to trade…
- Estimate an FFN layer's FLOPs, parameter and activation bytes, and the communication from sharding it. Which limit binds: compute, HBM or network?
Write the three costs down, then compare times. python d model, f hidden (often 4d), T tokens, bf16 flops = 2 (2 T d f) two matmuls wbytes = 2 d f 2 both…
- Estimate the GPU memory a very large matmul needs, then compare pipeline and tensor parallelism for it.
Count every live tensor. python Y = X @ W; X (m, k), W (k, n); bf16 = 2 bytes fwd = 2 (m k + k n + m n) m = k = n = 65536: 8 GiB each, 24 GiB total Training…