Mid-level interview questionAsked by candidates · interview write-up
You have two hours to build a working chatbot on the OpenAI API. Walk me through the message schema, session state, context-window strategy, streaming, and handling rate limits and retries.
Model answer
Messages are role and content pairs: system, user, assistant. Store each session's list plus a summary by session ID.
Scroll sideways to see the whole diagram
The context window is finite: the prompt and the reply share it.
Text version
- The context window is finite: the prompt and the reply share it.
- Part of the prompt is fixed. The rest, such as retrieved passages, is chosen for each request, and that is where the choices are.
- In this lesson's small help centre, all 14 articles come to 299 tokens; the article that answers "how do I stop paying every month" is 25. About 12 times the input for the same answer.
- Input tokens are billed and prefilled on every request. Spend them on passages that help.
Token counts: o200k_base on the 14 help articles of lib/labs/data/search-lab.json (lib/labs/context.test.ts). Cost and latency: AI Engineering Ch. 6, "RAG".
Before each call, count tokens: keep the system prompt, a summary of older turns and the latest turns, reserving room for the reply. Stream deltas to the client and save the assembled reply. On 429s and 5xx, retry with jittered exponential backoff, honour Retry-After, and cap attempts.
Practise the rest
12 Context selection questions, 492 in all, each with a model answer. Free.
More Context selection interview questions
- “Our model has a 200K-token window, so let's just send the whole knowledge base.” What's wrong with that?
- You rank passages with reciprocal rank fusion of BM25 and embeddings, and keep the top one. The passage that answers the query never reaches the model. How?
- Your pipeline concatenates retrieved passages in database order. Does it matter where the most relevant one lands?
- A long support conversation no longer fits the context window. Compare the ways to shrink it, and how each one fails.
- Your retriever's top passage is usually on topic but often contradicts the right answer. What's going on, and what do you change?
- What is context engineering, and how is it different from prompt engineering?