Memory bandwidth-bound
Also called bandwidth-bound, memory-bound
Work whose time is set by how fast data moves from memory to the arithmetic units; more memory bandwidth, or fewer bytes to move, speeds it up.
Interview questions about Memory bandwidth-bound
- What's the difference between compute-bound and memory bandwidth-bound work, and which one is decode?
Compute-bound work is limited by arithmetic; bandwidth-bound work by how fast data moves from memory.
- How would you show, on a given machine, that decode is bandwidth-bound and prefill compute-bound?
Measure both ceilings, a large memory copy and a large matrix multiply, and place each phase on a roofline.
- Your LLM training and serving is slow. Reason about memory-bound versus compute-bound kernels, the cost of materializing the attention matrix, HFU versus MFU, and which GPU profiling signals you'd read.
Compare each kernel's arithmetic intensity with the GPU's ratio of compute to bandwidth.
- Explain FlashAttention, the KV cache and RoPE, and the memory and compute trade-offs each addresses.
FlashAttention computes exact attention in tiles kept in on-chip SRAM, with an online softmax, so the full score matrix never goes to HBM.
- Explain how a server like vLLM runs a decoder-only model efficiently.
A decoder-only model runs prefill, then one decode step per token, and decode is memory-bound, so the win is batching many requests.
- How would you optimise large-scale training and inference on GPUs?
Profile first, then attack the binding limit. Training: BF16 or FP8 mixed precision, FlashAttention and fused kernels, activation checkpointing to trade…
- Explain the optimisations an ML compiler makes, such as kernel fusion, and how they map to the hardware.
A compiler turns the model graph into fewer, faster kernels. Kernel fusion matters most: chaining elementwise ops, norms or attention into one kernel keeps…
- Walk me through how a model goes from framework code to running on a GPU.
In eager PyTorch, each op dispatches to a kernel, often from cuBLAS or cuDNN, and the CPU launches it asynchronously on a CUDA stream.