# Where the time goes: bandwidth, batching and memory interview questions

> 10 where the time goes: bandwidth, batching and memory interview questions for AI and ML engineer roles, each with a model answer.

Canonical: https://llmdojo.dev/interview/topic/where-the-time-goes
Last updated: 2026-10-23

10 questions: 1 junior, 3 mid-level and 6 senior. Taught in [Where the Time Goes: Memory, Compute and Batching](https://llmdojo.dev/learn/inference-optimization/where-the-time-goes).

## [What's the difference between compute-bound and memory bandwidth-bound work, and which one is decode?](https://llmdojo.dev/interview/q/what-s-the-difference-between-compute-bound-and-memory-bandwidth-bound-work)

Level: Junior

Compute-bound work is limited by arithmetic; bandwidth-bound work by how fast data moves from memory. **Arithmetic intensity**, operations per byte read, decides which. Decode is bandwidth-bound: each step **reads every weight** for one token.

**Diagram: How much work each weight read buys**

1. Every forward pass has to bring the model's weights from memory to the arithmetic units: about 498 MB for GPT-2 small in 32-bit floats.
2. A decode step does that whole read to produce one token for one sequence, plus that sequence's KV cache. About 2 operations per weight, 4 bytes per weight: half an operation per byte.
3. Decode 8 sequences together and the same read serves all 8. Only the KV caches grow with the batch.
4. Prefill pushes every prompt token through in one pass, so one read of the weights serves all 960.
5. Operations per byte decides the limit. Few, and the chip waits on memory; many, and it waits on arithmetic. The next pages measure both on one laptop.

GPT-2 small (124,439,808 weights, float32); KV cache from the Module 01 formula at a 128-token context.

GPT-2 small in 4-byte floats reads **497.8 MB per token**, about half an operation per byte.

## [How would you show, on a given machine, that decode is bandwidth-bound and prefill compute-bound?](https://llmdojo.dev/interview/q/how-would-you-show-on-a-given-machine-that-decode-is-bandwidth)

Level: Mid-level

**Measure both ceilings**, a large memory copy and a large matrix multiply, and place each phase on a **roofline**. On our test laptop's CPU, a batch-1 decode step moved 507 MB in 9.1 ms: 55.7 GB/s, 46% of the copy ceiling, at about 0.5 operations per byte, far left of the ridge at 16. A 960-token prefill reached 51% of the matrix-multiply ceiling. For decode, **more bandwidth or fewer bytes** helps; more FLOP/s doesn't.

## [Why does decoding a batch of 8 cost far less than 8 separate decode steps, and why can't the batch grow forever?](https://llmdojo.dev/interview/q/why-does-decoding-a-batch-of-8-cost-far-less-than-8)

Level: Mid-level

Each step **reads the weights once** and uses them for every sequence; only each sequence's KV cache is extra. On our test laptop's Apple GPU, a step took 6.3 ms for one sequence and 9.3 ms for 8, so throughput rose from **159 to 860 tokens/s**.

Limits: the longer step **slows each user**, and every conversation's KV cache needs memory.

## [You're sizing an 80 GB accelerator for Llama 2 13B in 16-bit with 2,048-token conversations. How many fit, and what would you change if that's not enough?](https://llmdojo.dev/interview/q/you-re-sizing-an-80-gb-accelerator-for-llama-2-13b-in)

Level: Senior

**Weights plus KV cache** per conversation times conversations must fit. Weights take about 26 GB. Each cache is 2 × 40 layers × 5,120 × 2,048 tokens × 2 bytes = **1.68 GB**, so the remaining 54 GB holds 32, matching AI Engineering's example. Real servers fit fewer: activations, the framework and fragmentation take memory too.

The cache is **linear in tokens and bytes** per number, so cap conversation length or store the cache in fewer bytes.

## [Your chat app's replies range from 8 to 256 tokens. Which batching strategy would you pick, and what did the simulation show?](https://llmdojo.dev/interview/q/your-chat-app-s-replies-range-from-8-to-256-tokens-which)

Level: Senior

**Continuous batching**. Static batching waits for a full batch and dynamic caps the wait, but both finish a batch together, so a short reply **waits for its longest neighbour**. Continuous batching, Orca's iteration-level scheduling, returns each reply when done and **refills its seat**.

**Diagram: Request-level versus continuous batching**

1. Two requests are decoded together: A needs 8 more tokens, B needs 64.
2. In request-level batching the batch finishes together. A's reply is ready after 8 steps but waits 56 more for B.
3. In continuous batching A is returned as soon as its last token is generated.
4. Its seat is filled by a waiting request at the very next step, so the batch stays full.
5. Orca calls this iteration-level scheduling: the batch is decided again before every step.

After AI Engineering, Ch. 9, "Batching", and Orca (Yu et al., OSDI 2022). Illustrative lengths.

Simulated from our measured Apple GPU steps (200 requests, at most 8 at once), continuous gave 343 tokens/s and a 22 ms median first token; static batches gave 299 tokens/s and a 5.3 s median wait.

## [Estimate an FFN layer's FLOPs, parameter and activation bytes, and the communication from sharding it. Which limit binds: compute, HBM or network?](https://llmdojo.dev/interview/q/estimate-an-ffn-layer-s-flops-parameter-and-activation-bytes-and-the)

Level: Senior

Write the three costs down, then compare times.

```python
# d model, f hidden (often 4d), T tokens, bf16
flops = 2 * (2 * T * d * f)      # two matmuls
w_bytes = 2 * d * f * 2          # both weights
act_bytes = T * (2 * d + f) * 2  # in, out, hidden
# tensor parallel over n GPUs: one all-reduce of T*d
comm = 2 * (n - 1) / n * T * d * 2   # bytes per GPU
t_compute = flops / n / peak_flops
t_hbm = (w_bytes / n + act_bytes) / hbm_bw
t_net = comm / link_bw
```

**Diagram: How much work each weight read buys**

1. Every forward pass has to bring the model's weights from memory to the arithmetic units: about 498 MB for GPT-2 small in 32-bit floats.
2. A decode step does that whole read to produce one token for one sequence, plus that sequence's KV cache. About 2 operations per weight, 4 bytes per weight: half an operation per byte.
3. Decode 8 sequences together and the same read serves all 8. Only the KV caches grow with the batch.
4. Prefill pushes every prompt token through in one pass, so one read of the weights serves all 960.
5. Operations per byte decides the limit. Few, and the chip waits on memory; many, and it waits on arithmetic. The next pages measure both on one laptop.

GPT-2 small (124,439,808 weights, float32); KV cache from the Module 01 formula at a 128-token context.

The largest time binds. With few tokens, as in decode, intensity is roughly T FLOPs per byte, below a GPU's FLOPs-to-bandwidth ratio, so **HBM binds**; big prefill batches turn **compute-bound**. Sharding divides compute and weights by n but not the all-reduce, so over slow inter-node links the **network binds**.

## [Explain the optimisations an ML compiler makes, such as kernel fusion, and how they map to the hardware.](https://llmdojo.dev/interview/q/explain-the-optimisations-an-ml-compiler-makes-such-as-kernel-fusion-and)

Level: Senior

A compiler turns the model graph into fewer, faster kernels. **Kernel fusion** matters most: chaining elementwise ops, norms or attention into one kernel keeps intermediates in registers or shared memory instead of round-tripping HBM, and most such ops are **memory-bound**. Others: constant folding, layout changes to what tensor cores want, lowering to vendor libraries, autotuning tile sizes per GPU, static memory planning, and CUDA graphs against launch overhead. Fitting the hardware means sizing tiles to SM resources and keeping matmuls in **tensor-core shapes** and precisions.

## [Walk me through how a model goes from framework code to running on a GPU.](https://llmdojo.dev/interview/q/walk-me-through-how-a-model-goes-from-framework-code-to-running)

Level: Mid-level

In eager PyTorch, each op dispatches to a kernel, often from cuBLAS or cuDNN, and the CPU **launches it asynchronously** on a CUDA stream. The GPU runs it as thread blocks across its SMs, reading HBM. A compiler like torch.compile instead captures a graph, fuses ops and generates Triton kernels; CUDA graphs replay launches with less CPU overhead. Small ops end up **launch- or memory-bound**, large matmuls compute-bound on tensor cores.

## [Your training GPUs sit partly idle waiting for data. How do you confirm that, and how would you design a data-preparation pipeline that keeps them fed?](https://llmdojo.dev/interview/q/your-training-gpus-sit-partly-idle-waiting-for-data-how-do-you)

Level: Senior

First prove it: a profiler shows each step waiting on the next batch, and GPU utilisation dips in a sawtooth. A quick test is to repeat one cached batch; if throughput jumps, **the input pipeline is the bottleneck**.

Then move work out of the hot path. Tokenise, filter and dedupe **offline**, writing packed, pre-tokenised sequences into large shards instead of many small files. At train time, stream shards with parallel workers, prefetching and pinned memory, shuffle by shard order plus a buffer, and make the loader **resumable** from a saved position.

## [Your LLM training and serving is slow. Reason about memory-bound versus compute-bound kernels, the cost of materializing the attention matrix, HFU versus MFU, and which GPU profiling signals you'd read.](https://llmdojo.dev/interview/q/your-llm-training-and-serving-is-slow-reason-about-memory-bound-versus)

Level: Senior

Compare each kernel's **arithmetic intensity** with the GPU's ratio of compute to bandwidth. Large matmuls are compute-bound; softmax, norms, elementwise ops and small-batch decode are memory-bound, so fuse them.

Naive attention writes the full sequence-by-sequence score matrix to HBM: quadratic memory and traffic. FlashAttention tiles it through on-chip SRAM instead.

**MFU** counts only the model's required FLOPs against peak; HFU also counts recomputation, so activation checkpointing raises HFU, not MFU.

Profile: timeline gaps for data loading, launch overhead and communication; per-kernel achieved bandwidth versus tensor-core utilization.

## Related
- [All interview questions](https://llmdojo.dev/interview)