Compute-bound
Work whose time is set by how much arithmetic it needs; a chip with more operations per second speeds it up.
Interview questions about Compute-bound
- What's the difference between compute-bound and memory bandwidth-bound work, and which one is decode?
Compute-bound work is limited by arithmetic; bandwidth-bound work by how fast data moves from memory.
- How would you show, on a given machine, that decode is bandwidth-bound and prefill compute-bound?
Measure both ceilings, a large memory copy and a large matrix multiply, and place each phase on a roofline.
- Your LLM training and serving is slow. Reason about memory-bound versus compute-bound kernels, the cost of materializing the attention matrix, HFU versus MFU, and which GPU profiling signals you'd read.
Compare each kernel's arithmetic intensity with the GPU's ratio of compute to bandwidth.
- Estimate an FFN layer's FLOPs, parameter and activation bytes, and the communication from sharding it. Which limit binds: compute, HBM or network?
Write the three costs down, then compare times. python d model, f hidden (often 4d), T tokens, bf16 flops = 2 (2 T d f) two matmuls wbytes = 2 d f 2 both…
- Walk me through how a model goes from framework code to running on a GPU.
In eager PyTorch, each op dispatches to a kernel, often from cuBLAS or cuDNN, and the CPU launches it asynchronously on a CUDA stream.