Chunking and indexing interview questions
6 questions with model answers
- junior 1
- mid 3
- senior 2
6 chunking and indexing questions asked in AI engineer, ML engineer and applied AI engineer interviews, each with a model answer you can say out loud in a minute or two: 1 junior, 3 mid-level and 2 senior. The lesson Chunking and Indexing teaches the ideas behind them.Updated
Model answer
Embedding models and generators have input limits, and one vector for a whole document blurs every topic in it.
Chunks give each idea its own vector, so a specific question finds the specific passage, and only that passage spends prompt tokens.
Model answer
Enough to keep whole what must stay whole. A span of L words always lands inside some chunk when overlap ≥ L − 1: for a 9-word answer, overlap 8.
The cost: more chunks to embed and store (13 instead of 7 in the course's example), and repeated text when neighbours are both retrieved.
Model answer
Recursive, usually: split by section, then paragraph, then sentence, until each piece fits. Cuts land where the author put them.
Fixed sizes ignore meaning, so whether a fact survives is luck. Some documents suggest their own unit: a question-and-answer pair, or a code-aware splitter.
Model answer
Give it its context. Cut from its document, it may never say what it's about: “It usually arrives in 5 to 7 business days” doesn't say “refund”.
Scroll sideways to see the whole diagram
Cut out of its document, a chunk can lose what it's about. This one never says “refund”.
Text version
- Cut out of its document, a chunk can lose what it's about. This one never says “refund”.
- The cheap fix is metadata: the document title, the section heading, tags, or entities such as error codes.
- Anthropic's contextual retrieval uses a model instead: given the whole document and the chunk, it writes a short context, usually 50–100 tokens, situating the chunk.
- The context is prepended to the chunk.
- Then the augmented chunk is indexed, so a search for “refund” can now find it.
Prepend the title and section, add keywords or entities, list questions it answers, or generate a short context with a model, as Anthropic's contextual retrieval does.
Model answer
By experiment. Smaller chunks let more distinct pieces fit (halve the size, fit twice as many) but split information and double the vectors to compute, store and search. Larger ones keep context together but carry more irrelevant text. Both are bounded by the embedding model's and generator's limits.
Scroll sideways to see the whole diagram
Smaller chunks mean more of them fit in the prompt, giving the model a wider range of information.
Text version
- Smaller chunks mean more of them fit in the prompt, giving the model a wider range of information.
- But information that spans a boundary can be lost: if only one half is retrieved, the model never sees the other.
- And computational overhead grows: AI Engineering notes that halving the chunk size doubles the chunks to index and the vectors to generate and store, and the search space.
- Larger chunks keep context together, but each one spends more of the budget, much of it on text the question doesn't need.
- A chunk can't exceed what the embedding model or the generator can read. Between those limits there's no universal best: experiment on your own queries.
The book says there's no universal best: measure retrieval on real questions with known answers at a few sizes.
Model answer
1 − recall@20: the share of relevant chunks missing from the top 20. Contextual embeddings cut it 35% (5.7% → 3.7%); adding contextual BM25, 49%; adding a reranker, 67% (to 1.9%), on their evaluation sets.
The cost is a model call per chunk at indexing time, repeated whenever documents change, plus longer chunks to embed and store.