Senior interview question
How do you choose a chunk size?
Model answer
By experiment. Smaller chunks let more distinct pieces fit (halve the size, fit twice as many) but split information and double the vectors to compute, store and search. Larger ones keep context together but carry more irrelevant text. Both are bounded by the embedding model's and generator's limits.
Scroll sideways to see the whole diagram
Smaller chunks mean more of them fit in the prompt, giving the model a wider range of information.
Text version
- Smaller chunks mean more of them fit in the prompt, giving the model a wider range of information.
- But information that spans a boundary can be lost: if only one half is retrieved, the model never sees the other.
- And computational overhead grows: AI Engineering notes that halving the chunk size doubles the chunks to index and the vectors to generate and store, and the search space.
- Larger chunks keep context together, but each one spends more of the budget, much of it on text the question doesn't need.
- A chunk can't exceed what the embedding model or the generator can read. Between those limits there's no universal best: experiment on your own queries.
The book says there's no universal best: measure retrieval on real questions with known answers at a few sizes.
Practise the rest
6 Chunking questions, 492 in all, each with a model answer. Free.