Mid-level interview questionAsked at Decagon · interview write-up
Design a chatbot for a single enterprise customer and domain that holds multi-turn conversations and answers only from that customer's approved documents.
Model answer
RAG over the approved corpus only, versioned so removed documents disappear. Hybrid retrieval plus a reranker.
Scroll sideways to see the whole diagram
Search your own documents for passages related to the question, with keywords, embeddings or both.
Text version
- Search your own documents for passages related to the question, with keywords, embeddings or both.
- A reranker reads the question with each candidate and keeps the few that actually answer it.
- The kept passages go into the prompt, with instructions to answer only from them.
- The model answers from evidence in its context instead of memory alone, so the answer can be checked against its sources.
The pattern of Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, as used in production systems.
For multi-turn, rewrite each follow-up into a standalone query using the history, and keep a short conversation summary. The prompt says answer only from the passages and cite them; when support is weak, the bot abstains or hands off to a human. Evaluate on real customer questions: faithfulness, correct abstention and resolution rate.
Practise the rest
22 RAG questions, 492 in all, each with a model answer. Free.
More from Decagon interview questions.
More RAG interview questions
- What is retrieval-augmented generation, and how does it reduce hallucination?
- How do you choose a chunking strategy for RAG?
- Why combine keyword and embedding search, and where does a reranker fit?
- How do you evaluate a RAG pipeline?
- Your RAG chatbot keeps giving wrong answers. How do you find out why, and fix it?
- Design a RAG system that has to meet strict latency and cost targets while serving many tenants.