Utilization
Also called utilisation
The share of a resource's capacity actually in use. A rented machine that sits idle still bills by the hour, so low utilization raises the cost per request.
Interview questions about Utilization
- Design request batching for GPU inference, including scheduling, autoscaling and routing.
Batch at the iteration level: with continuous batching, finished sequences leave and queued ones join every decode step, so short requests never wait for long…
- Your training GPUs sit partly idle waiting for data. How do you confirm that, and how would you design a data-preparation pipeline that keeps them fed?
First prove it: a profiler shows each step waiting on the next batch, and GPU utilisation dips in a sawtooth.
- Your LLM training and serving is slow. Reason about memory-bound versus compute-bound kernels, the cost of materializing the attention matrix, HFU versus MFU, and which GPU profiling signals you'd read.
Compare each kernel's arithmetic intensity with the GPU's ratio of compute to bandwidth.
- Design a shared distributed ML training platform with team quotas, coordinated worker placement, versioned data, experiment tracking and checkpoint recovery.
Jobs are declarative specs: image, data version, resources. The scheduler gives each team a guaranteed quota and lets idle capacity be borrowed, preemptibly.
- Design a job scheduler for ML training and batch inference on a shared GPU cluster.
Jobs enter a queue with a priority and a team quota, with fair share across teams.