# Selecting, ordering and compressing context interview questions

> 12 selecting, ordering and compressing context interview questions for AI and ML engineer roles, each with a model answer.

Canonical: https://llmdojo.dev/interview/topic/context
Last updated: 2026-10-23

12 questions: 1 junior, 6 mid-level and 5 senior. Taught in [Choosing Context: Selection, Ordering and Compression](https://llmdojo.dev/learn/prompt-context-engineering/context-selection).

## [“Our model has a 200K-token window, so let's just send the whole knowledge base.” What's wrong with that?](https://llmdojo.dev/interview/q/our-model-has-a-200k-token-window-so-let-s-just-send)

Level: Junior

**Fitting isn't helping**. Every input token is **billed and prefilled on every request**, and a long context isn't necessarily used well. In the lesson's help centre, all 14 articles are 299 tokens against 25 for the one that answers, **about 12 times the input**; with thousands of documents, the ratio grows.

## [You rank passages with reciprocal rank fusion of BM25 and embeddings, and keep the top one. The passage that answers the query never reaches the model. How?](https://llmdojo.dev/interview/q/you-rank-passages-with-reciprocal-rank-fusion-of-bm25-and-embeddings-and)

Level: Mid-level

Fusion **rewards agreement, not correctness**: each retriever gives 1 / (k + rank), k = 60 being typical.

**Diagram: From candidates to a packed context**

1. Retrievers propose candidates. BM25 returns only passages that share a word with the query; embedding search always returns a nearest neighbour.
2. Candidates are ranked by one retriever's score, or by fusing the two rankings (reciprocal rank fusion).
3. Selection is a cut: the top k that fit the budget go in. A good passage ranked below the cut is lost, however relevant.
4. Then the order. Ranking for a search page and ordering for a model are different jobs: the model reads all of it, but not equally well everywhere.

After AI Engineering Ch. 6, "RAG": "Combining retrieval algorithms" and "Reranking"; Liu et al. 2023 (arXiv:2307.03172), Section 5.

For "how do I stop paying every month", BM25 returned only the Wi-Fi article, through the word "every"; embeddings ranked it third. It scored 1/61 + 1/63 and beat the cancellation article's 1/61, so the answer, ranked #2, **was cut**. Judge selection on **queries with known answers**, and leave k room.

## [Your pipeline concatenates retrieved passages in database order. Does it matter where the most relevant one lands?](https://llmdojo.dev/interview/q/your-pipeline-concatenates-retrieved-passages-in-database-order-does-it-matter-where)

Level: Mid-level

**It can**. In Liu et al., GPT-3.5-Turbo with 20 passages scored 75.8% with the answer first, **53.8% at position 10**, below its 56.1% closed-book score, and 63.2% last. Claude-1.3 was much flatter, and the best end varied by model. These are 2023 models, so **sort by relevance** (best first, or best next to the question) and compare both on a needle-style test.

## [A long support conversation no longer fits the context window. Compare the ways to shrink it, and how each one fails.](https://llmdojo.dev/interview/q/a-long-support-conversation-no-longer-fits-the-context-window-compare-the)

Level: Senior

**Every method loses something**; you choose what. Dropping the oldest turns is simple, but the book warns it "can be fatally wrong": in an illustrative 40-turn chat, keeping the last 10 loses turn 1, which account and plan the user is cancelling. A model-written summary keeps more, but costs a call and is new text, so it can **omit or misstate details**. Truncating, or routing to a larger-window model, are the blunt options. **Write down what must survive** (goal, IDs, numbers, constraints) and test it early, late and buried.

## [Your retriever's top passage is usually on topic but often contradicts the right answer. What's going on, and what do you change?](https://llmdojo.dev/interview/q/your-retriever-s-top-passage-is-usually-on-topic-but-often-contradicts)

Level: Senior

Retrieval **scores similarity**, not whether a passage answers. For "is this toy safe for a two-year-old", BM25, embeddings and their fusion all put "safe for children over three" above "not safe for children under three", the one that answers; embeddings barely register "not". With top 1 the model gets only the misleading passage. Evaluate retrieval against questions with known answers, **keep more than one candidate**, and **rerank** with a more precise second stage.

## [What is context engineering, and how is it different from prompt engineering?](https://llmdojo.dev/interview/q/what-is-context-engineering-and-how-is-it-different-from-prompt-engineering)

Level: Mid-level

Prompt engineering is wording the instruction. **Context engineering** decides **everything else** in the window on each call: retrieved passages, history, tool results, examples, and their order.

**Diagram: From candidates to a packed context**

1. Retrievers propose candidates. BM25 returns only passages that share a word with the query; embedding search always returns a nearest neighbour.
2. Candidates are ranked by one retriever's score, or by fusing the two rankings (reciprocal rank fusion).
3. Selection is a cut: the top k that fit the budget go in. A good passage ranked below the cut is lost, however relevant.
4. Then the order. Ranking for a search page and ordering for a model are different jobs: the model reads all of it, but not equally well everywhere.

After AI Engineering Ch. 6, "RAG": "Combining retrieval algorithms" and "Reranking"; Liu et al. 2023 (arXiv:2307.03172), Section 5.

The window is a **budget**, so the work is selecting what's relevant, compressing what's long, and placing what matters where the model attends best.

## [Design LLM search or RAG for very long documents and very long user queries.](https://llmdojo.dev/interview/q/design-llm-search-or-rag-for-very-long-documents-and-very-long)

Level: Senior

Treat the context window as a **budget**. For long documents, chunk along their structure and index chunks plus document summaries, so retrieval goes coarse to fine. For long queries, have a model rewrite them into focused sub-queries, retrieve for each, then merge and rerank.

**Diagram: One window, many claims on it**

1. The context window is finite: the prompt and the reply share it.
2. Part of the prompt is fixed. The rest, such as retrieved passages, is chosen for each request, and that is where the choices are.
3. In this lesson's small help centre, all 14 articles come to 299 tokens; the article that answers "how do I stop paying every month" is 25. About 12 times the input for the same answer.
4. Input tokens are billed and prefilled on every request. Spend them on passages that help.

Token counts: o200k_base on the 14 help articles of lib/labs/data/search-lab.json (lib/labs/context.test.ts). Cost and latency: AI Engineering Ch. 6, "RAG".

Pack the best chunks up to a token budget, with the most relevant at the start or end, since models use the **middle of long contexts** worst. When the answer needs a whole document, use **map-reduce**: extract per chunk, then answer over the extracts.

## [Your model needs to handle much longer sequences. How would you optimise attention for that?](https://llmdojo.dev/interview/q/your-model-needs-to-handle-much-longer-sequences-how-would-you-optimise)

Level: Senior

First find which cost hurts: attention compute **grows quadratically** with length and dominates long prefills, while the KV cache grows linearly and caps decode batch size.

Exact fixes first. **FlashAttention** tiles the work in on-chip memory, never storing the n × n score matrix: same result, memory linear in n. For the cache: grouped-query attention, quantisation, paged allocation.

Then approximations that change the model: **sliding-window** or sparse patterns like Longformer's local plus global attention, or linear-attention and state-space layers. Past one GPU, ring attention splits the sequence across devices. Re-check long-context quality.

## [Design per-user memory so a chatbot can personalise its answers over time.](https://llmdojo.dev/interview/q/design-per-user-memory-so-a-chatbot-can-personalise-its-answers-over)

Level: Senior

Keep two stores per user: a small **profile** of stable facts and preferences, always in the prompt, and episode summaries retrieved by relevance.

**Diagram: One window, many claims on it**

1. The context window is finite: the prompt and the reply share it.
2. Part of the prompt is fixed. The rest, such as retrieved passages, is chosen for each request, and that is where the choices are.
3. In this lesson's small help centre, all 14 articles come to 299 tokens; the article that answers "how do I stop paying every month" is 25. About 12 times the input for the same answer.
4. Input tokens are billed and prefilled on every request. Spend them on passages that help.

Token counts: o200k_base on the 14 help articles of lib/labs/data/search-lab.json (lib/labs/context.test.ts). Cost and latency: AI Engineering Ch. 6, "RAG".

After each conversation, a model **extracts atomic facts** with a timestamp and source; newer facts override conflicting ones, and duplicates merge. At query time, fit the profile plus the top retrieved memories into a fixed token budget.

The hard part is trust: users can **see and delete** memories, sensitive data needs consent and retention limits, stores are isolated per user, and remembered text is data, never instructions. Measure whether it helps.

## [Rapid fire: how do RAG and fine-tuning relate, what do you do when content exceeds the context window, how do you keep conversation state, and how do you manage prompts?](https://llmdojo.dev/interview/q/rapid-fire-how-do-rag-and-fine-tuning-relate-what-do-you)

Level: Mid-level

**RAG and fine-tuning** fill different gaps: RAG supplies changing or citable facts, fine-tuning shapes format, style or a narrow skill. **Over the window**: retrieve only the relevant chunks, or summarise in a map-reduce pass. **Conversation state**: keep recent turns verbatim, summarise older ones, and store durable facts outside the prompt. Prompts: version them like code, with templates and a regression suite. Treat retrieved text as data, never instructions, to limit prompt injection.

## [Build the message-history layer for a conversation agent: fetch the history page by page, then compact it whenever it exceeds the model's token limit.](https://llmdojo.dev/interview/q/build-the-message-history-layer-for-a-conversation-agent-fetch-the-history)

Level: Mid-level

```python
def load_history(fetch_page):
    msgs, cursor = [], None
    while True:
        page = fetch_page(cursor)
        msgs.extend(page["messages"])
        cursor = page.get("next_cursor")
        if not cursor:
            return msgs

def compact(msgs, limit, count, summarize, keep=6):
    if sum(count(m) for m in msgs) <= limit:
        return msgs
    while True:
        cut = max(len(msgs) - keep, 0)
        note = {"role": "system",
                "content": summarize(msgs[:cut])}
        out = [note] + msgs[cut:]
        if sum(count(m) for m in out) <= limit:
            return out
        if keep == 0:
            raise ValueError("summary exceeds limit")
        keep //= 2
```

Follow the cursor until none comes back, retrying failed pages. Compact only when over the limit: summarise older turns into one note, keep the **recent turns verbatim**, and halve that window if it still doesn't fit. Count with the model's own tokenizer, and never separate a tool call from its result.

**Diagram: Three ways to shrink a context, and what each loses**

1. An illustrative long conversation. The most important sentence is the first one.
2. Dropping the oldest turns is simple and common. It assumes early turns matter least, which is wrong exactly here.
3. Truncating to fit a smaller model's limit keeps one end and silently discards the other.
4. A summary keeps more, but it is another model call, and what it leaves out is decided by a model you haven't checked.
5. Each method throws something away. Decide what must survive, and test with cases that depend on it.

After AI Engineering Ch. 6, "Memory" and Ch. 10, "Step 3. Add Model Router and Gateway" > "Router". Example text is illustrative, not a model output.

## [You have two hours to build a working chatbot on the OpenAI API. Walk me through the message schema, session state, context-window strategy, streaming, and handling rate limits and retries.](https://llmdojo.dev/interview/q/you-have-two-hours-to-build-a-working-chatbot-on-the-openai)

Level: Mid-level

Messages are role and content pairs: system, user, assistant. Store each session's list plus a summary by session ID.

**Diagram: One window, many claims on it**

1. The context window is finite: the prompt and the reply share it.
2. Part of the prompt is fixed. The rest, such as retrieved passages, is chosen for each request, and that is where the choices are.
3. In this lesson's small help centre, all 14 articles come to 299 tokens; the article that answers "how do I stop paying every month" is 25. About 12 times the input for the same answer.
4. Input tokens are billed and prefilled on every request. Spend them on passages that help.

Token counts: o200k_base on the 14 help articles of lib/labs/data/search-lab.json (lib/labs/context.test.ts). Cost and latency: AI Engineering Ch. 6, "RAG".

Before each call, **count tokens**: keep the system prompt, a summary of older turns and the latest turns, reserving room for the reply. **Stream** deltas to the client and save the assembled reply. On 429s and 5xx, retry with **jittered exponential backoff**, honour Retry-After, and cap attempts.

## Related
- [All interview questions](https://llmdojo.dev/interview)