Mid-level interview questionAsked at Sierra · interview write-up
Build the message-history layer for a conversation agent: fetch the history page by page, then compact it whenever it exceeds the model's token limit.
Model answer
def load_history(fetch_page):
msgs, cursor = [], None
while True:
page = fetch_page(cursor)
msgs.extend(page["messages"])
cursor = page.get("next_cursor")
if not cursor:
return msgs
def compact(msgs, limit, count, summarize, keep=6):
if sum(count(m) for m in msgs) <= limit:
return msgs
while True:
cut = max(len(msgs) - keep, 0)
note = {"role": "system",
"content": summarize(msgs[:cut])}
out = [note] + msgs[cut:]
if sum(count(m) for m in out) <= limit:
return out
if keep == 0:
raise ValueError("summary exceeds limit")
keep //= 2
Follow the cursor until none comes back, retrying failed pages. Compact only when over the limit: summarise older turns into one note, keep the recent turns verbatim, and halve that window if it still doesn't fit. Count with the model's own tokenizer, and never separate a tool call from its result.
Scroll sideways to see the whole diagram
An illustrative long conversation. The most important sentence is the first one.
Text version
- An illustrative long conversation. The most important sentence is the first one.
- Dropping the oldest turns is simple and common. It assumes early turns matter least, which is wrong exactly here.
- Truncating to fit a smaller model's limit keeps one end and silently discards the other.
- A summary keeps more, but it is another model call, and what it leaves out is decided by a model you haven't checked.
- Each method throws something away. Decide what must survive, and test with cases that depend on it.
After AI Engineering Ch. 6, "Memory" and Ch. 10, "Step 3. Add Model Router and Gateway" > "Router". Example text is illustrative, not a model output.
Practise the rest
12 Context selection questions, 492 in all, each with a model answer. Free.
More from Sierra interview questions.
More Context selection interview questions
- “Our model has a 200K-token window, so let's just send the whole knowledge base.” What's wrong with that?
- You rank passages with reciprocal rank fusion of BM25 and embeddings, and keep the top one. The passage that answers the query never reaches the model. How?
- Your pipeline concatenates retrieved passages in database order. Does it matter where the most relevant one lands?
- A long support conversation no longer fits the context window. Compare the ways to shrink it, and how each one fails.
- Your retriever's top passage is usually on topic but often contradicts the right answer. What's going on, and what do you change?
- What is context engineering, and how is it different from prompt engineering?