Chunking and Indexing

You will be able to

• Explain why documents are chunked, and which model limits bound chunk size.
• Chunk by fixed size in characters, words or tokens, and show how boundaries break facts.
• Size overlap to the length of the text that must stay whole, and count what it costs.
• Split documents recursively along their own structure (sections, paragraphs, sentences).
• Weigh smaller against larger chunks and choose a size by experiment.
• Add context to chunks (metadata, entities, questions, generated context) so they can be found.
• Rewrite conversational queries so they stand on their own, and decline to guess when they can't be resolved.

Topics

1. 1Why split documents
2. 2Fixed-size chunks
3. 3Overlap
4. 4Split along the document's structure
5. 5How big should a chunk be?
6. 6Give each chunk its context
7. 7Rewriting queries, and wrap-up