Senior interview questionAsked at OpenAI · interview write-up
Design a RAG system that has to meet strict latency and cost targets while serving many tenants.
Model answer
Start by splitting the latency budget across retrieval, reranking and generation. I'd use hybrid retrieval, BM25 plus dense vectors, with a tenant filter applied inside the search, never after it. A small cross-encoder reranks, and only the best few chunks enter the prompt, which keeps prefill short.
Scroll sideways to see the whole diagram
Search your own documents for passages related to the question, with keywords, embeddings or both.
Text version
- Search your own documents for passages related to the question, with keywords, embeddings or both.
- A reranker reads the question with each candidate and keeps the few that actually answer it.
- The kept passages go into the prompt, with instructions to answer only from them.
- The model answers from evidence in its context instead of memory alone, so the answer can be checked against its sources.
The pattern of Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, as used in production systems.
For cost: cache embeddings and answers per tenant, keep the system prompt a stable prefix for prompt caching, stream tokens, and route easy questions to a smaller model. Track recall and p95 per tenant, with rate limits so no tenant starves the rest.
Practise the rest
22 RAG questions, 492 in all, each with a model answer. Free.
More from OpenAI interview questions.
More RAG interview questions
- What is retrieval-augmented generation, and how does it reduce hallucination?
- How do you choose a chunking strategy for RAG?
- Why combine keyword and embedding search, and where does a reranker fit?
- How do you evaluate a RAG pipeline?
- Your RAG chatbot keeps giving wrong answers. How do you find out why, and fix it?
- Design an agent for searching internal documents that respects permissions, cites sources and abstains when it should.