Topic 5
Checking citations
The step after retrieval is what Hands-On Large Language Models (Ch. 8, "From Search to RAG") calls grounded generation: the model answers from the retrieved passages, and preferably cites them. Citations let a reader check the answer. They also need checking themselves.
Liu, Zhang and Liang (2023), "Evaluating Verifiability in Generative Search Engines", audited four generative search engines: Bing Chat, NeevaAI, Perplexity and YouChat. They measured two things for every response:
- Citation recall: is every statement fully supported by its citations?
- Citation precision: does each citation support the statement it's attached to?
Predict first
The lab shows a five-sentence answer to a refund question, written by hand to contain each kind of problem. Click through the sentences.
Checking a cited answer
Illustrative answer · one sentence counted as one claim
Question: “How do I get a refund, and how long does it take?” Click a sentence.
- [1] You can ask for your money back within 30 days of purchase. Open Settings, then Billing, then Request refund.
- [2] The money goes back to the card you paid with. It usually arrives in 5 to 7 business days.
- [3] To cancel, open Settings, then Billing, then Cancel plan. You keep access until the end of the billing period.
Cites passage 2, which says 5 to 7 days: the citation doesn't support it.
Citation recall
40%
Citation precision
50%
Faithfulness
60%
Three numbers, three different questions:
- Citation recall 40%: only 2 of 5 sentences are supported by what they cite.
- Citation precision 50%: 2 of the 4 citations support their sentence.
- Faithfulness 60%: 3 of 5 sentences are supported by some retrieved passage, cited or not (next page).
Sentence 4 is true but cites the wrong passage: faithful, yet a reader who follows the citation can't verify it. Sentence 3 cites the right passage and contradicts it: the most dangerous kind, because the citation makes it look checked.
Quick check