Mid-level interview questionAsked at OpenAI · interview write-up
Design how a chatbot falls back when it doesn't know the answer, covering uncertainty, retrieval and tools.
Model answer
Make abstaining a valid output. Questions needing fresh or private facts go to retrieval or a tool; if nothing clears a relevance threshold, say so rather than guess. Then check answers: require citations to retrieved passages, and treat disagreement across several samples as an uncertainty signal. Low confidence falls back to a clarifying question or a human. Evaluate on unanswerable questions as well as answerable ones.
Practise the rest
11 Hallucination questions, 492 in all, each with a model answer. Free.
More from OpenAI interview questions.
More Hallucination interview questions
- What is a hallucination, and why can a next-token predictor produce one so fluently?
- GPT-2 small completes 'The capital of Australia is the city of' with Sydney. Walk me through why a model does that.
- Why do long generated answers leave more room for hallucination than one-word answers?
- You want to automatically flag answers that are probably wrong, using the model's token probabilities. How would you design and validate that?
- You're shipping a support bot that answers from your company's policy documents. What do you put in place to reduce hallucination, and how do you evaluate it?
- How would you measure an LLM's answer quality and detect hallucinations against the context it was given? Then implement the evaluation component.