Senior interview questionAsked at Cohere · interview write-up
Design an enterprise research assistant where every claim in the answer carries a citation the user can verify.
Model answer
Make citation part of generation, not decoration. Retrieve with hybrid search and a reranker, respecting document permissions, and give each passage an id. Have the model write claim by claim, each with the supporting span as structured output.
Then verify every claim: an entailment model or judge checks it against its cited span, and unsupported claims are dropped or flagged. Show citations as quotes that open the source. Evaluate citation precision and recall: is each claim supported, and is each citation needed?
Scroll sideways to see the whole diagram
Search your own documents for passages related to the question, with keywords, embeddings or both.
Text version
- Search your own documents for passages related to the question, with keywords, embeddings or both.
- A reranker reads the question with each candidate and keeps the few that actually answer it.
- The kept passages go into the prompt, with instructions to answer only from them.
- The model answers from evidence in its context instead of memory alone, so the answer can be checked against its sources.
The pattern of Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, as used in production systems.
Practise the rest
22 RAG questions, 492 in all, each with a model answer. Free.
More from Cohere interview questions.
More RAG interview questions
- What is retrieval-augmented generation, and how does it reduce hallucination?
- How do you choose a chunking strategy for RAG?
- Why combine keyword and embedding search, and where does a reranker fit?
- How do you evaluate a RAG pipeline?
- Your RAG chatbot keeps giving wrong answers. How do you find out why, and fix it?
- Design a RAG system that has to meet strict latency and cost targets while serving many tenants.