Senior interview questionAsked by candidates · collected on GitHub
You're shipping a support bot that answers from your company's policy documents. What do you put in place to reduce hallucination, and how do you evaluate it?
Model answer
Ground answers in retrieved policy text. Context steers strongly: in our toy, one true sentence moved " Canberra" from 0.0436 to 0.8248, though that mostly measures copying, and retrieval can find the wrong text.
So: retrieve, require a cited passage for each answer and check it supports the claim, keep answers short and restricted to the given context, and let the bot say it doesn't know. Evaluate local factual consistency: is each answer supported by the documents it was given? That's far easier to check than truth against world knowledge.
Practise the rest
11 Hallucination questions, 492 in all, each with a model answer. Free.
More Hallucination interview questions
- What is a hallucination, and why can a next-token predictor produce one so fluently?
- GPT-2 small completes 'The capital of Australia is the city of' with Sydney. Walk me through why a model does that.
- Why do long generated answers leave more room for hallucination than one-word answers?
- You want to automatically flag answers that are probably wrong, using the model's token probabilities. How would you design and validate that?
- Design how a chatbot falls back when it doesn't know the answer, covering uncertainty, retrieval and tools.
- How would you measure an LLM's answer quality and detect hallucinations against the context it was given? Then implement the evaluation component.