Your training GPUs sit partly idle waiting for data. How do you confirm that, and how would you design a data-preparation pipeline that keeps them fed?

First prove it: a profiler shows each step waiting on the next batch, and GPU utilisation dips in a sawtooth. A quick test is to repeat one cached batch; if throughput jumps, the input pipeline is the bottleneck.

Then move work out of the hot path. Tokenise, filter and dedupe offline, writing packed, pre-tokenised sequences into large shards instead of many small files. At train time, stream shards with parallel workers, prefetching and pinned memory, shuffle by shard order plus a buffer, and make the loader resumable from a saved position.