Retrieval-augmented generation interview questions

Before the model answers, retrieve relevant passages from your documents and put them in the prompt, so it answers from evidence in its context, not memory alone.

It cuts hallucination when retrieval finds the right passage and the prompt says to answer only from it, cite it, and admit when it isn't there.

Chunks should be small enough to be specific and big enough to stand alone. Split on the document's own structure first, like headings and paragraphs, then cap the size in tokens with a little overlap so no sentence loses its context. Prefix each chunk with its title or section path. Then pick the size by measuring retrieval on real questions, not by rule of thumb.

Embeddings catch paraphrases; keyword search such as BM25 catches exact terms like product codes, names and error messages. Hybrid search runs both and merges the rankings, for example with reciprocal rank fusion. A reranker, a cross-encoder that reads the query and a passage together, then rescores the top few dozen: too slow for the whole corpus, much sharper on a shortlist.

Score the two halves separately. Retrieval: on questions with known source passages, does the right one land in the top k? Generation: is the answer faithful to the retrieved passages, and does it answer the question? Use a judge model checked against human labels, on a fixed set run after every change. Bad retrieval and bad generation need different fixes.

Sort failures by stage before changing anything: log the retrieved passages for every bad answer. Right passage missing: fix retrieval with better chunking, hybrid search, query rewriting, metadata filters, or more candidates plus a reranker. Passage present, answer wrong: fix generation by putting it first, trimming distractors, and asking the model to quote and to abstain. Source wrong or stale: fix the data. Then add each failure to the eval set, so the fix stays fixed.

Start by splitting the latency budget across retrieval, reranking and generation. I'd use hybrid retrieval, BM25 plus dense vectors, with a tenant filter applied inside the search, never after it. A small cross-encoder reranks, and only the best few chunks enter the prompt, which keeps prefill short.

For cost: cache embeddings and answers per tenant, keep the system prompt a stable prefix for prompt caching, stream tokens, and route easy questions to a smaller model. Track recall and p95 per tenant, with rate limits so no tenant starves the rest.

Permissions first: store each chunk with its source's access-control list and filter by the user's identity inside the retrieval call, so the model never sees what the user can't open. Sync ACL changes fast; stale permissions are a leak.

The agent plans searches, reformulates and opens documents, but every answer must cite passages, and a checker verifies claims against them. With no relevant evidence, or conflicting evidence, it abstains and suggests owners. Evaluate recall, citation accuracy, correct abstention, and permission tests with users of different access.

Offline: parse documents, chunk on their structure, embed, and index chunks with metadata and permissions in a vector and keyword index, updated incrementally.

Online: rewrite the query, run hybrid search filtered by the user's access, rerank, and pack the best passages into a prompt that says to cite them and abstain when the answer isn't there.

Around it: an eval set scoring retrieval and faithfulness separately, logged passages, caching, and a per-stage latency budget.

Retrieval becomes tools the agent calls, a search per source, so it can reformulate and search again. Grounding: answers cite passages, a checker flags claims the passages don't support, and the agent abstains when evidence is missing.

Access control lives in retrieval, not the prompt: filter every search by the user's permissions, and run tools with that user's credentials and least privilege. Retrieved text is untrusted, a vector for indirect injection, so write actions need confirmation. Log every retrieval and action for audit, and evaluate retrieval, faithfulness and leakage separately.

Index more than files: parsed symbols with signatures and docstrings, the import and call graph, tests, docs and recent diffs. Chunk on syntax boundaries, a whole function with its path and class, not fixed windows.

Rank with several signals: exact identifier search, embeddings for intent, and graph expansion from symbols in the open file or stack trace, then a reranker. Cut noise by skipping vendored, generated and build files. Measure recall of the files the real fix touched, then task success: do the tests pass?

Turn each asset into a searchable card: a table's name, columns and comments, tags, owner, sample values and lineage; a notebook's title, cell summary, and the tables it reads and writes.

Retrieve with exact match on table and column names plus embeddings on descriptions, filtered by access permissions first. Rerank with a cross-encoder and usage signals: query popularity, freshness, certified status, and lineage closeness to tables already in the session. Evaluate on logged requests: recall at k of the table the user really queried, then whether the SQL runs correctly.

Queries mix exact identifiers, like a table name or catalogue path, with intent, like churn by region. Run a keyword index over names, paths and column names, and embeddings over descriptions and comments.

Normalise identifiers first, splitting snake_case and camelCase so cust_id matches customer id. Merge them with reciprocal rank fusion, but let an exact name match go straight to the top. Filter by permissions first, and tune on search logs.

Make citation part of generation, not decoration. Retrieve with hybrid search and a reranker, respecting document permissions, and give each passage an id. Have the model write claim by claim, each with the supporting span as structured output.

Then verify every claim: an entailment model or judge checks it against its cited span, and unsupported claims are dropped or flagged. Show citations as quotes that open the source. Evaluate citation precision and recall: is each claim supported, and is each citation needed?

Ingest: parse documents to text, keep metadata like source, date and permissions, and split by structure into chunks of a few hundred tokens. Embed them in batches and store vectors with the metadata.

At query time, embed the question with the same model, retrieve the top k with filters, and give the chat model chunks with ids to cite. Make retrieval a tool, so the agent can search again with a rewritten query.

Sparse retrieval like BM25 matches tokens, so its tokenization matters: words with stemming suit English prose, while code, part numbers and other languages need subwords or character n-grams. Dense retrieval embeds meaning, so it catches paraphrases but misses rare exact strings. A hybrid fuses both rankings, say with reciprocal rank fusion, then reranks the top. Evaluate on labelled queries with recall at k and NDCG, broken down by query type, against each retriever alone.

import numpy as np

def chunk(text, size=200, overlap=40):
    w = text.split()
    step = size - overlap
    stop = max(len(w) - overlap, 1)
    return [" ".join(w[i:i + size])
            for i in range(0, stop, step)]

def build_index(chunks, embed):
    E = np.array([embed(c) for c in chunks], float)
    return E / np.linalg.norm(E, axis=1, keepdims=True)

def retrieve(q, E, embed, k=4):
    v = np.asarray(embed(q), float)
    s = E @ (v / np.linalg.norm(v))
    top = np.argsort(-s)[:k]
    return top, s[top]

def hit_rate(results, gold):
    hits = [bool(set(r) & g) for r, g in zip(results, gold)]
    return sum(hits) / len(hits)

Normalise embeddings so retrieval is a dot product. Evaluate on questions labelled with their relevant chunk ids: hit rate or recall at k, plus MRR. The prompt wraps chunks as quoted evidence, says nothing inside them is an instruction, and requires a chunk id per claim. Delimiters reduce injection without stopping it, so the model gets no tools. If the top score falls below a threshold tuned on the eval set, abstain.

Ingest memos with the metadata lawyers filter on, such as matter, client, jurisdiction and date, plus ethical-wall permissions enforced inside the retrieval query. Chunk along the memo's structure, issue, analysis, conclusion, keeping the heading path, so citations land on a paragraph. Use hybrid search, since case names and statute numbers need exact match, then a reranker.

The model answers only from retrieved passages with pinpoint citations, and a verifier checks each claim against its cited span. Flag memos the law may have overtaken, and abstain on conflicting evidence.

Index policies as versioned documents with metadata: region, product, user type, effective dates. Extract the case facts from the ticket first, retrieve policies whose metadata matches, then run an applicability check: does each candidate cover this region, date and booking type? The model writes numbered steps, each citing a policy clause, or says none applies.

Log policy versions, retrieved chunks and prompt version with every answer. When an agent reports a wrong instruction, replay it against that exact policy set to see whether retrieval, applicability or generation failed.

Ingest through the product APIs and webhooks so edits and deletions arrive fast. Chunk pages by heading; treat an issue as summary, description and comments, with project, status and assignee as metadata for filters. Hybrid search, since issue keys need exact match, then a reranker boosting recent items. Store each item's permissions and filter by user inside the query, re-syncing when access changes. Answers cite links; evaluate on real employee questions.

Index chunks with metadata and their source's access-control list, kept in sync. At query time, rewrite the query, run BM25 and dense retrieval with the permission filter inside both, fuse with reciprocal rank fusion, then a cross-encoder reranks the top few dozen. Measure each stage on labelled queries: recall at k for candidates, NDCG and MRR for the final order, then answer faithfulness and permission-leak tests.

RAG over the approved corpus only, versioned so removed documents disappear. Hybrid retrieval plus a reranker.

For multi-turn, rewrite each follow-up into a standalone query using the history, and keep a short conversation summary. The prompt says answer only from the passages and cite them; when support is weak, the bot abstains or hands off to a human. Evaluate on real customer questions: faithfulness, correct abstention and resolution rate.

Stages, each tracked per document: detect type, OCR scans while keeping page and box positions with confidence, extract metadata and keywords, chunk, embed, and write to a hybrid index with access-control fields.

Each stage is idempotent, keyed by document hash and stage version, and persists its output, so a failure retries only that stage with backoff, and poison documents land in a dead-letter queue. Reprocessing is a version bump: a new OCR model reruns affected stages, then the index swaps over.

Queries: hybrid retrieval, metadata filters, a reranker. Monitor OCR confidence and search quality.