Senior interview questionAsked at NVIDIA · interview write-up
How would you optimise large-scale training and inference on GPUs?
Model answer
Profile first, then attack the binding limit. Training: BF16 or FP8 mixed precision, FlashAttention and fused kernels, activation checkpointing to trade compute for memory, ZeRO or FSDP sharding, and parallelism laid out to the topology, tensor parallel inside NVLink, pipeline and data across nodes, with communication overlapped. Track MFU.
Inference: decode is memory-bound, so cut bytes with quantised weights and KV cache and GQA, raise batch size with paged KV and continuous batching, use speculative decoding for latency, and add tensor parallelism only as far as latency needs.
Practise the rest
17 Latency & throughput questions, 492 in all, each with a model answer. Free.
More from NVIDIA interview questions.
More Latency & throughput interview questions
- A chat reply streams in token by token. What two waits does the user feel, and which one does streaming make matter most?
- How do you get TTFT, inter-token latency, TPOT and total latency from one streamed reply?
- Why report p50 and p99 latency rather than the mean?
- You time a GPU request by reading the clock right after calling the model and get 0.3 ms per token. What went wrong, and what else can silently skew a timing?
- Server logs show a TTFT of 80 ms, yet users of your reasoning assistant say it takes seconds to start answering. Explain the gap.
- Your team wants to raise the batch size to cut serving cost. What happens to users, and which number would you use to decide?