Topic 4
What counts as a term
Before any scoring, the query and the documents are broken into terms. AI Engineering (Ch. 6, "Term-based retrieval") calls this tokenization, and it's a different step from the model tokenizers of Module 01: the goal is units that should match, not units a model can read.
Common choices: lowercase everything, remove punctuation, and drop stop words like “the”, “and” and “is”. Term-based search engines usually do this for you.
Predict first
Scroll sideways to see the whole diagram
Term-based retrieval only sees the words, so how text becomes terms decides what can match.
Text version
- Term-based retrieval only sees the words, so how text becomes terms decides what can match.
- This lab's tokenizer lowercases and splits on punctuation, so “two-year-old” becomes three terms.
- Stop words go. What's left includes “two”, which matches the two-factor article: a match on spelling, not meaning. Treating common phrases as single terms (“hot dog”) is one fix.
Splitting can destroy meaning. The book's example is “hot dog”: as “hot” and “dog”, neither word means the food. One fix is to treat common n-grams as single terms, so the bigram “hot dog” is indexed as one term.
The same choices cut both ways for identifiers. This tokenizer keeps 0x80070005 whole, so a query for that error code finds its article with a high score. A tokenizer that split it into “0” and “80070005” would also match any stray “0” in other articles.
Quick check