How RAG works interview questions
6 questions with model answers
- junior 1
- mid 3
- senior 2
6 how rag works questions asked in AI engineer, ML engineer and applied AI engineer interviews, each with a model answer you can say out loud in a minute or two: 1 junior, 3 mid-level and 2 senior. The lesson How RAG Works: Retrieve, Then Generate teaches the ideas behind them.Updated
Model answer
A retriever finds the passages most relevant to a question in your own data, and a generator answers with them in its prompt.
Scroll sideways to see the whole diagram
Search your own documents for passages related to the question, with keywords, embeddings or both.
Text version
- Search your own documents for passages related to the question, with keywords, embeddings or both.
- A reranker reads the question with each candidate and keeps the few that actually answer it.
- The kept passages go into the prompt, with instructions to answer only from them.
- The model answers from evidence in its context instead of memory alone, so the answer can be checked against its sources.
The pattern of Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, as used in production systems.
It builds context per query, so answers can use current, per-user data and cite where they came from.
Model answer
Maybe not. Anthropic's guidance: under about 200,000 tokens you can put the whole knowledge base in the prompt.
The trade-off is per-query cost and how well the model uses a long context: sending 150,000 tokens instead of 5 chunks of 300 is 100× the input on every request. Large or fast-changing data needs retrieval.
Model answer
The index belongs to the retriever. Re-index when the chunking, the embedding model, or the tokenizer you chunk by changes.
Swapping the generator doesn't require it: it reads whatever text retrieval hands it.
Model answer
Usually retrieval. A generator only helps on the questions where it has the passage.
Illustratively, with 90% accuracy given the passage and 20% without: start at 0.69; a 95% generator gives 0.725; 90% retrieval gives 0.83. With perfect retrieval, the generator's accuracy is the ceiling.
Model answer
Route before retrieving:
Scroll sideways to see the whole diagram
Before searching, decide whether the question needs a search at all, and which source can answer it.
Text version
- Before searching, decide whether the question needs a search at all, and which source can answer it.
- Some messages need no retrieval: answering them directly saves a search and keeps irrelevant text out of the prompt.
- Questions about how things work go to the document index.
- Questions about numbers in a table (“how many units sold last week?”) need a SQL query written from the schema and executed, not a text search.
- Whatever the route, the generator answers from what came back.
- No search for messages like “thanks”: a search always returns something, which then competes for attention.
- Documents for how things work.
- Tables for numbers: text-to-SQL from the schema, run it, answer from the result; with many tables, first predict which ones.
- Multi-query or multi-hop when one search can't answer it.
A wrong route fails before any ranking happens, so evaluate the router too.
Model answer
When the knowledge fits comfortably in the prompt and rarely changes, a retriever adds work without adding much. When the answer must be computed, like a sum over a sales table, text search can't find it: use SQL. And when failures are about format or behaviour rather than missing facts, retrieval won't fix them.
Long contexts don't make RAG obsolete, though: data outgrows windows, and long contexts aren't always used well.