# Retrieval-augmented generation interview questions > 22 retrieval-augmented generation interview questions for AI and ML engineer roles, each with a model answer. Canonical: https://llmdojo.dev/interview/topic/rag Last updated: 2026-10-23 22 questions: 1 junior, 11 mid-level and 10 senior. ## [What is retrieval-augmented generation, and how does it reduce hallucination?](https://llmdojo.dev/interview/q/what-is-retrieval-augmented-generation-and-how-does-it-reduce-hallucination) Level: Junior Before the model answers, **retrieve** relevant passages from your documents and put them in the prompt, so it answers from **evidence in its context**, not memory alone. **Diagram: Retrieval-augmented generation** 1. Search your own documents for passages related to the question, with keywords, embeddings or both. 2. A reranker reads the question with each candidate and keeps the few that actually answer it. 3. The kept passages go into the prompt, with instructions to answer only from them. 4. The model answers from evidence in its context instead of memory alone, so the answer can be checked against its sources. The pattern of Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, as used in production systems. It cuts hallucination when retrieval finds the right passage and the prompt says to **answer only from it**, cite it, and admit when it isn't there. ## [How do you choose a chunking strategy for RAG?](https://llmdojo.dev/interview/q/how-do-you-choose-a-chunking-strategy-for-rag) Level: Mid-level Chunks should be **small enough to be specific** and big enough to stand alone. Split on the document's own structure first, like headings and paragraphs, then cap the size in tokens with a little overlap so no sentence loses its context. Prefix each chunk with its **title or section path**. Then pick the size by **measuring retrieval** on real questions, not by rule of thumb. ## [Why combine keyword and embedding search, and where does a reranker fit?](https://llmdojo.dev/interview/q/why-combine-keyword-and-embedding-search-and-where-does-a-reranker-fit) Level: Mid-level Embeddings catch **paraphrases**; keyword search such as BM25 catches **exact terms** like product codes, names and error messages. Hybrid search runs both and merges the rankings, for example with reciprocal rank fusion. A **reranker**, a cross-encoder that reads the query and a passage together, then rescores the top few dozen: too slow for the whole corpus, much sharper on a shortlist. ## [How do you evaluate a RAG pipeline?](https://llmdojo.dev/interview/q/how-do-you-evaluate-a-rag-pipeline) Level: Mid-level Score the two halves separately. **Retrieval**: on questions with known source passages, does the right one land in the top k? **Generation**: is the answer **faithful** to the retrieved passages, and does it answer the question? Use a judge model checked against human labels, on a fixed set run after every change. Bad retrieval and bad generation need **different fixes**. ## [Your RAG chatbot keeps giving wrong answers. How do you find out why, and fix it?](https://llmdojo.dev/interview/q/your-rag-chatbot-keeps-giving-wrong-answers-how-do-you-find-out) Level: Senior Sort failures by stage before changing anything: log the retrieved passages for every bad answer. **Right passage missing**: fix retrieval with better chunking, hybrid search, query rewriting, metadata filters, or more candidates plus a reranker. **Passage present, answer wrong**: fix generation by putting it first, trimming distractors, and asking the model to quote and to abstain. **Source wrong or stale**: fix the data. Then add each failure to the **eval set**, so the fix stays fixed. ## [Design a RAG system that has to meet strict latency and cost targets while serving many tenants.](https://llmdojo.dev/interview/q/design-a-rag-system-that-has-to-meet-strict-latency-and-cost) Level: Senior Start by splitting the latency budget across retrieval, reranking and generation. I'd use **hybrid retrieval**, BM25 plus dense vectors, with a **tenant filter applied inside the search**, never after it. A small cross-encoder reranks, and only the best few chunks enter the prompt, which keeps prefill short. **Diagram: Retrieval-augmented generation** 1. Search your own documents for passages related to the question, with keywords, embeddings or both. 2. A reranker reads the question with each candidate and keeps the few that actually answer it. 3. The kept passages go into the prompt, with instructions to answer only from them. 4. The model answers from evidence in its context instead of memory alone, so the answer can be checked against its sources. The pattern of Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, as used in production systems. For cost: cache embeddings and answers per tenant, keep the system prompt a stable prefix for prompt caching, stream tokens, and route easy questions to a smaller model. Track recall and p95 per tenant, with rate limits so no tenant starves the rest. ## [Design an agent for searching internal documents that respects permissions, cites sources and abstains when it should.](https://llmdojo.dev/interview/q/design-an-agent-for-searching-internal-documents-that-respects-permissions-cites-sources) Level: Senior Permissions first: store each chunk with its source's **access-control list** and filter by the user's identity inside the retrieval call, so the model never sees what the user can't open. Sync ACL changes fast; stale permissions are a leak. The agent plans searches, reformulates and opens documents, but every answer must **cite passages**, and a checker verifies claims against them. With no relevant evidence, or conflicting evidence, it **abstains** and suggests owners. Evaluate recall, citation accuracy, correct abstention, and permission tests with users of different access. ## [Design a production RAG system end to end, from ingesting documents to serving answers.](https://llmdojo.dev/interview/q/design-a-production-rag-system-end-to-end-from-ingesting-documents-to) Level: Mid-level **Offline**: parse documents, chunk on their structure, embed, and index chunks with metadata and permissions in a vector and keyword index, updated incrementally. **Diagram: Retrieval-augmented generation** 1. Search your own documents for passages related to the question, with keywords, embeddings or both. 2. A reranker reads the question with each candidate and keeps the few that actually answer it. 3. The kept passages go into the prompt, with instructions to answer only from them. 4. The model answers from evidence in its context instead of memory alone, so the answer can be checked against its sources. The pattern of Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, as used in production systems. **Online**: rewrite the query, run hybrid search filtered by the user's access, rerank, and pack the best passages into a prompt that says to cite them and abstain when the answer isn't there. Around it: an **eval set** scoring retrieval and faithfulness separately, logged passages, caching, and a per-stage latency budget. ## [Design a RAG-based agent system: how it retrieves, how it grounds its answers, and how it enforces access control.](https://llmdojo.dev/interview/q/design-a-rag-based-agent-system-how-it-retrieves-how-it-grounds) Level: Senior Retrieval becomes **tools** the agent calls, a search per source, so it can reformulate and search again. **Grounding**: answers cite passages, a checker flags claims the passages don't support, and the agent abstains when evidence is missing. **Diagram: Indirect injection: the attacker writes the data** 1. The attacker never talks to the model. They put text where a model will later read it. 2. The user is not the attacker. They ask an ordinary question. 3. Retrieval or a tool call brings the planted text into the context, next to the user's request. 4. Nothing in the token stream says the text is data, so the model may treat it as an instruction. 5. The action runs with whatever tools and data the application gave the model. That is where the damage is done. After Greshake et al. (arXiv:2302.12173) and AI Engineering, Ch. 5, "Indirect prompt injection". Illustrative scenario. **Access control lives in retrieval**, not the prompt: filter every search by the user's permissions, and run tools with that user's credentials and least privilege. Retrieved text is untrusted, a vector for indirect injection, so write actions need confirmation. Log every retrieval and action for audit, and evaluate retrieval, faithfulness and leakage separately. ## [You're designing retrieval for a coding agent. What do you index, how do you chunk and rank, how do you keep noise out, and how do you measure it?](https://llmdojo.dev/interview/q/you-re-designing-retrieval-for-a-coding-agent-what-do-you-index) Level: Senior Index more than files: parsed **symbols** with signatures and docstrings, the import and call graph, tests, docs and recent diffs. Chunk on syntax boundaries, a whole function with its path and class, not fixed windows. Rank with several signals: exact identifier search, embeddings for intent, and graph expansion from symbols in the open file or stack trace, then a reranker. Cut noise by skipping vendored, generated and build files. Measure recall of the files the real fix touched, then **task success**: do the tests pass? ## [A coding agent writes SQL and notebooks over a company's data platform. How would you retrieve and rerank the right tables and notebooks for a request?](https://llmdojo.dev/interview/q/a-coding-agent-writes-sql-and-notebooks-over-a-company-s-data) Level: Senior Turn each asset into a **searchable card**: a table's name, columns and comments, tags, owner, sample values and lineage; a notebook's title, cell summary, and the tables it reads and writes. Retrieve with exact match on table and column names plus embeddings on descriptions, **filtered by access permissions** first. Rerank with a cross-encoder and usage signals: query popularity, freshness, certified status, and lineage closeness to tables already in the session. Evaluate on logged requests: recall at k of the table the user really queried, then whether the SQL runs correctly. ## [Users search an enterprise catalogue of tables, columns and dashboards by exact name and by meaning. How would you design hybrid retrieval over that metadata?](https://llmdojo.dev/interview/q/users-search-an-enterprise-catalogue-of-tables-columns-and-dashboards-by-exact) Level: Mid-level Queries mix **exact identifiers**, like a table name or catalogue path, with intent, like churn by region. Run a keyword index over names, paths and column names, and embeddings over descriptions and comments. Normalise identifiers first, splitting snake_case and camelCase so cust_id matches customer id. Merge them with **reciprocal rank fusion**, but let an exact name match go straight to the top. Filter by permissions first, and tune on search logs. ## [Design an enterprise research assistant where every claim in the answer carries a citation the user can verify.](https://llmdojo.dev/interview/q/design-an-enterprise-research-assistant-where-every-claim-in-the-answer-carries) Level: Senior Make citation part of generation, not decoration. Retrieve with hybrid search and a reranker, respecting document permissions, and give each passage an id. Have the model write claim by claim, each with the **supporting span** as structured output. Then **verify every claim**: an entailment model or judge checks it against its cited span, and unsupported claims are dropped or flagged. Show citations as quotes that open the source. Evaluate citation precision and recall: is each claim supported, and is each citation needed? **Diagram: Retrieval-augmented generation** 1. Search your own documents for passages related to the question, with keywords, embeddings or both. 2. A reranker reads the question with each candidate and keeps the few that actually answer it. 3. The kept passages go into the prompt, with instructions to answer only from them. 4. The model answers from evidence in its context instead of memory alone, so the answer can be checked against its sources. The pattern of Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, as used in production systems. ## [Using the Mistral API, how would you build a RAG agent end to end: ingestion, chunking, embeddings and retrieval?](https://llmdojo.dev/interview/q/using-the-mistral-api-how-would-you-build-a-rag-agent-end) Level: Mid-level Ingest: parse documents to text, keep metadata like source, date and permissions, and split by structure into chunks of a few hundred tokens. Embed them in batches and store vectors with the metadata. At query time, embed the question with **the same model**, retrieve the top k with filters, and give the chat model chunks with ids to cite. Make retrieval a **tool**, so the agent can search again with a rewritten query. ## [Compare tokenization choices and dense versus sparse retrieval for search and RAG, then design and evaluate a hybrid retriever.](https://llmdojo.dev/interview/q/compare-tokenization-choices-and-dense-versus-sparse-retrieval-for-search-and-rag) Level: Mid-level Sparse retrieval like BM25 matches tokens, so **its tokenization matters**: words with stemming suit English prose, while code, part numbers and other languages need subwords or character n-grams. Dense retrieval embeds meaning, so it catches paraphrases but misses **rare exact strings**. A hybrid fuses both rankings, say with reciprocal rank fusion, then reranks the top. Evaluate on labelled queries with recall at k and NDCG, **broken down by query type**, against each retriever alone. **Diagram: Retrieval-augmented generation** 1. Search your own documents for passages related to the question, with keywords, embeddings or both. 2. A reranker reads the question with each candidate and keeps the few that actually answer it. 3. The kept passages go into the prompt, with instructions to answer only from them. 4. The model answers from evidence in its context instead of memory alone, so the answer can be checked against its sources. The pattern of Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, as used in production systems. ## [Build a minimal embedding-based RAG pipeline: chunk, index, retrieve and assemble the prompt. How do you evaluate retrieval, resist injected instructions and abstain when unsure?](https://llmdojo.dev/interview/q/build-a-minimal-embedding-based-rag-pipeline-chunk-index-retrieve-and-assemble) Level: Mid-level ```python def chunk(text, size=200, overlap=40): w = text.split() step = size - overlap stop = max(len(w) - overlap, 1) return [" ".join(w[i:i + size]) for i in range(0, stop, step)] def build_index(chunks, embed): E = np.array([embed(c) for c in chunks], float) return E / np.linalg.norm(E, axis=1, keepdims=True) def retrieve(q, E, embed, k=4): v = np.asarray(embed(q), float) s = E @ (v / np.linalg.norm(v)) top = np.argsort(-s)[:k] return top, s[top] def hit_rate(results, gold): hits = [bool(set(r) & g) for r, g in zip(results, gold)] return sum(hits) / len(hits) ``` Normalise embeddings so retrieval is a dot product. Evaluate on questions labelled with their relevant chunk ids: **hit rate or recall at k**, plus MRR. The prompt wraps chunks as quoted evidence, says nothing inside them is an instruction, and requires a chunk id per claim. Delimiters reduce injection without stopping it, so the model gets no tools. If the top score falls below a threshold tuned on the eval set, **abstain**. ## [Design a question-answering agent over a law firm's legal memos, from ingestion and chunking to grounded, cited answers at scale.](https://llmdojo.dev/interview/q/design-a-question-answering-agent-over-a-law-firm-s-legal-memos) Level: Senior Ingest memos with the metadata lawyers filter on, such as matter, client, jurisdiction and date, plus **ethical-wall permissions** enforced inside the retrieval query. Chunk along the memo's structure, issue, analysis, conclusion, keeping the heading path, so citations land on a paragraph. Use hybrid search, since case names and statute numbers need exact match, then a reranker. The model answers only from retrieved passages with **pinpoint citations**, and a verifier checks each claim against its cited span. Flag memos the law may have overtaken, and abstain on conflicting evidence. ## [Design a RAG system that reads company policies and writes step-by-step, cited instructions for a human support agent. How do you check a policy applies, and debug a wrong instruction?](https://llmdojo.dev/interview/q/design-a-rag-system-that-reads-company-policies-and-writes-step-by) Level: Senior Index policies as **versioned documents** with metadata: region, product, user type, effective dates. Extract the case facts from the ticket first, retrieve policies whose metadata matches, then run an **applicability check**: does each candidate cover this region, date and booking type? The model writes numbered steps, each citing a policy clause, or says none applies. Log policy versions, retrieved chunks and prompt version with every answer. When an agent reports a wrong instruction, **replay it** against that exact policy set to see whether retrieval, applicability or generation failed. ## [Design an MVP RAG assistant that answers employee questions across Jira issues and Confluence pages, covering ingestion, retrieval, ranking and permissions.](https://llmdojo.dev/interview/q/design-an-mvp-rag-assistant-that-answers-employee-questions-across-jira-issues) Level: Mid-level Ingest through the product APIs and webhooks so edits and deletions arrive fast. Chunk pages by heading; treat an issue as summary, description and comments, with project, status and assignee as **metadata for filters**. Hybrid search, since issue keys need exact match, then a reranker boosting recent items. Store each item's **permissions** and filter by user inside the query, re-syncing when access changes. Answers cite links; evaluate on real employee questions. ## [Design the retrieval and ranking pipeline behind a RAG assistant, including indexing, access control and the metrics you'd evaluate it with.](https://llmdojo.dev/interview/q/design-the-retrieval-and-ranking-pipeline-behind-a-rag-assistant-including-indexing) Level: Mid-level Index chunks with metadata and their source's **access-control list**, kept in sync. At query time, rewrite the query, run BM25 and dense retrieval with the permission filter inside both, fuse with reciprocal rank fusion, then a cross-encoder **reranks** the top few dozen. Measure each stage on labelled queries: recall at k for candidates, NDCG and MRR for the final order, then answer faithfulness and **permission-leak tests**. ## [Design a chatbot for a single enterprise customer and domain that holds multi-turn conversations and answers only from that customer's approved documents.](https://llmdojo.dev/interview/q/design-a-chatbot-for-a-single-enterprise-customer-and-domain-that-holds) Level: Mid-level RAG over the **approved corpus only**, versioned so removed documents disappear. Hybrid retrieval plus a reranker. **Diagram: Retrieval-augmented generation** 1. Search your own documents for passages related to the question, with keywords, embeddings or both. 2. A reranker reads the question with each candidate and keeps the few that actually answer it. 3. The kept passages go into the prompt, with instructions to answer only from them. 4. The model answers from evidence in its context instead of memory alone, so the answer can be checked against its sources. The pattern of Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, as used in production systems. For multi-turn, **rewrite each follow-up** into a standalone query using the history, and keep a short conversation summary. The prompt says answer only from the passages and cite them; when support is weak, the bot **abstains or hands off** to a human. Evaluate on real customer questions: faithfulness, correct abstention and resolution rate. ## [Design an AI-powered document search pipeline with OCR, metadata and keyword extraction, indexing, retries and reprocessing.](https://llmdojo.dev/interview/q/design-an-ai-powered-document-search-pipeline-with-ocr-metadata-and-keyword) Level: Senior Stages, each tracked per document: detect type, **OCR** scans while keeping page and box positions with confidence, extract metadata and keywords, chunk, embed, and write to a **hybrid index** with access-control fields. Each stage is idempotent, keyed by document hash and stage version, and persists its output, so a failure retries only that stage with backoff, and poison documents land in a dead-letter queue. **Reprocessing** is a version bump: a new OCR model reruns affected stages, then the index swaps over. Queries: hybrid retrieval, metadata filters, a reranker. Monitor OCR confidence and search quality. ## Related - [All interview questions](https://llmdojo.dev/interview)