Topic 7
Saying “I don't know”
Some questions have no answer in your documents. The right response is to say so. That's abstention, and RAG systems are built in a way that works against it.
Predict first
Scroll sideways to see the whole diagram
Some questions have no answer in your documents. Embedding retrieval returns k chunks anyway.
Text version
- Some questions have no answer in your documents. Embedding retrieval returns k chunks anyway.
- One guard: a minimum similarity, so weak matches never reach the prompt. Thresholds are model-specific and need tuning on real queries.
- Another: instruct the generator to answer only from the passages and to say when they don't contain the answer.
- Both have to be tested: on questions you know are unanswerable, and on answerable ones, so the system doesn't start refusing those.
Two guards, used together:
- Filter weak matches. Hands-On Large Language Models (Ch. 8, "Caveats of dense retrieval") suggests a maximum distance (minimum similarity) for relevance. Thresholds are specific to the embedding model, so set them from real queries, not intuition.
- Instruct the generator. Tell it to answer only from the passages and to say when they don't contain the answer. The RAG pipeline diagram's last step says exactly this.
Then measure it. Add questions you know are unanswerable to the test set, and count how often the system abstains. And keep counting abstentions on answerable questions: a system tuned to refuse whenever it's unsure stops hallucinating and starts being useless. Error analysis in Module 04 found “refused an in-scope question” as its own failure mode.
Quick check