Topic 6
Faithfulness and answer relevance
Hands-On Large Language Models (Ch. 8, "RAG Evaluation") notes that RAG evaluation is still developing, that human evaluation is preferred, and that libraries such as Ragas automate it with a model acting as judge. Two of the metrics it highlights:
- Faithfulness: whether the answer is consistent with the provided context.
- Answer relevance: how relevant the answer is to the question.
The Ragas documentation defines faithfulness precisely: the number of claims in the response supported by the retrieved context, divided by the total number of claims. It ranges from 0 to 1.
Predict first
Scroll sideways to see the whole diagram
Split the answer into the separate factual claims it makes.
Text version
- Split the answer into the separate factual claims it makes.
- Check each claim against the retrieved passages. With a model as judge, all of Module 04's warnings about judges apply.
- The score is the supported fraction. It measures whether the answer stuck to its evidence, not whether the evidence was correct or relevant.
Steps as described in the Ragas documentation for the faithfulness metric.
That's why faithfulness is one metric among several, not the score. Pair it with:
- answer relevance, because an answer can be faithful to the passages and still not answer the question;
- retrieval metrics, because faithfulness to the wrong passages is no help;
- end-to-end correctness against references where you have them (scoring against references).
And because these metrics usually rely on a model as judge, everything in using a model as a judge applies: write the judge prompt carefully, and check its verdicts against people before trusting the numbers.
Quick check