Arithmetic intensity
Also called operations per byte
Operations done per byte read from memory. Low intensity means bandwidth is the limit; high intensity means compute is.
Interview questions about Arithmetic intensity
- What's the difference between compute-bound and memory bandwidth-bound work, and which one is decode?
Compute-bound work is limited by arithmetic; bandwidth-bound work by how fast data moves from memory.
- How would you show, on a given machine, that decode is bandwidth-bound and prefill compute-bound?
Measure both ceilings, a large memory copy and a large matrix multiply, and place each phase on a roofline.
- Your LLM training and serving is slow. Reason about memory-bound versus compute-bound kernels, the cost of materializing the attention matrix, HFU versus MFU, and which GPU profiling signals you'd read.
Compare each kernel's arithmetic intensity with the GPU's ratio of compute to bandwidth.