Mid-level interview questionAsked at Cohere · interview write-up
Users ask about events after the model's training cutoff. How should it recognise those questions and say so?
Model answer
The model doesn't know today's date, so tell it: put the date and the training cutoff in the system prompt.
Then train the behaviour. Fine-tune on questions about things that change, like recent events or latest versions, with answers that give what held at the cutoff and flag it may be stale, or call search. Include stable questions so it doesn't over-refuse. Evaluate on dated questions either side of the cutoff.
Practise the rest
11 Hallucination questions, 492 in all, each with a model answer. Free.
More from Cohere interview questions.
More Hallucination interview questions
- What is a hallucination, and why can a next-token predictor produce one so fluently?
- GPT-2 small completes 'The capital of Australia is the city of' with Sydney. Walk me through why a model does that.
- Why do long generated answers leave more room for hallucination than one-word answers?
- You want to automatically flag answers that are probably wrong, using the model's token probabilities. How would you design and validate that?
- You're shipping a support bot that answers from your company's policy documents. What do you put in place to reduce hallucination, and how do you evaluate it?
- Design how a chatbot falls back when it doesn't know the answer, covering uncertainty, retrieval and tools.