Mid-level interview question
GPT-2 small completes 'The capital of Australia is the city of' with Sydney. Walk me through why a model does that.
Model answer
Because likely text isn't true text. In our run, " Sydney" was the top token at 0.216 and " Canberra" only fifth at 0.0436, and greedy stated Sydney with no hedge.
There's no lookup: knowledge is spread through the weights, and a plausible wrong answer, the best-known city, can outweigh the right one. We didn't measure why Sydney wins; one plausible reason is that it appears near "Australia" in far more web text.
Practise the rest
11 Hallucination questions, 492 in all, each with a model answer. Free.
More Hallucination interview questions
- What is a hallucination, and why can a next-token predictor produce one so fluently?
- Why do long generated answers leave more room for hallucination than one-word answers?
- You want to automatically flag answers that are probably wrong, using the model's token probabilities. How would you design and validate that?
- You're shipping a support bot that answers from your company's policy documents. What do you put in place to reduce hallucination, and how do you evaluate it?
- Design how a chatbot falls back when it doesn't know the answer, covering uncertainty, retrieval and tools.
- How would you measure an LLM's answer quality and detect hallucinations against the context it was given? Then implement the evaluation component.