Mid-level interview question
Why do long generated answers leave more room for hallucination than one-word answers?
Model answer
Every generated token becomes context for the next, and nothing marks it as the model's own guess, so each step is another chance to go wrong. AI Engineering describes snowballing: the model keeps justifying an early wrong assumption.
A right start doesn't protect the rest: GPT-2's greedy Everest answer began correctly, then put the summit at about 1,000 feet. That's one reason concise answers seem to help.
Practise the rest
11 Hallucination questions, 492 in all, each with a model answer. Free.
More Hallucination interview questions
- What is a hallucination, and why can a next-token predictor produce one so fluently?
- GPT-2 small completes 'The capital of Australia is the city of' with Sydney. Walk me through why a model does that.
- You want to automatically flag answers that are probably wrong, using the model's token probabilities. How would you design and validate that?
- You're shipping a support bot that answers from your company's policy documents. What do you put in place to reduce hallucination, and how do you evaluate it?
- Design how a chatbot falls back when it doesn't know the answer, covering uncertainty, retrieval and tools.
- How would you measure an LLM's answer quality and detect hallucinations against the context it was given? Then implement the evaluation component.