Selecting, ordering and compressing context interview questions

Fitting isn't helping. Every input token is billed and prefilled on every request, and a long context isn't necessarily used well. In the lesson's help centre, all 14 articles are 299 tokens against 25 for the one that answers, about 12 times the input; with thousands of documents, the ratio grows.

Fusion rewards agreement, not correctness: each retriever gives 1 / (k + rank), k = 60 being typical.

For "how do I stop paying every month", BM25 returned only the Wi-Fi article, through the word "every"; embeddings ranked it third. It scored 1/61 + 1/63 and beat the cancellation article's 1/61, so the answer, ranked #2, was cut. Judge selection on queries with known answers, and leave k room.

It can. In Liu et al., GPT-3.5-Turbo with 20 passages scored 75.8% with the answer first, 53.8% at position 10, below its 56.1% closed-book score, and 63.2% last. Claude-1.3 was much flatter, and the best end varied by model. These are 2023 models, so sort by relevance (best first, or best next to the question) and compare both on a needle-style test.

Every method loses something; you choose what. Dropping the oldest turns is simple, but the book warns it "can be fatally wrong": in an illustrative 40-turn chat, keeping the last 10 loses turn 1, which account and plan the user is cancelling. A model-written summary keeps more, but costs a call and is new text, so it can omit or misstate details. Truncating, or routing to a larger-window model, are the blunt options. Write down what must survive (goal, IDs, numbers, constraints) and test it early, late and buried.

Retrieval scores similarity, not whether a passage answers. For "is this toy safe for a two-year-old", BM25, embeddings and their fusion all put "safe for children over three" above "not safe for children under three", the one that answers; embeddings barely register "not". With top 1 the model gets only the misleading passage. Evaluate retrieval against questions with known answers, keep more than one candidate, and rerank with a more precise second stage.

Prompt engineering is wording the instruction. Context engineering decides everything else in the window on each call: retrieved passages, history, tool results, examples, and their order.

The window is a budget, so the work is selecting what's relevant, compressing what's long, and placing what matters where the model attends best.

Treat the context window as a budget. For long documents, chunk along their structure and index chunks plus document summaries, so retrieval goes coarse to fine. For long queries, have a model rewrite them into focused sub-queries, retrieve for each, then merge and rerank.

Pack the best chunks up to a token budget, with the most relevant at the start or end, since models use the middle of long contexts worst. When the answer needs a whole document, use map-reduce: extract per chunk, then answer over the extracts.

First find which cost hurts: attention compute grows quadratically with length and dominates long prefills, while the KV cache grows linearly and caps decode batch size.

Exact fixes first. FlashAttention tiles the work in on-chip memory, never storing the n × n score matrix: same result, memory linear in n. For the cache: grouped-query attention, quantisation, paged allocation.

Then approximations that change the model: sliding-window or sparse patterns like Longformer's local plus global attention, or linear-attention and state-space layers. Past one GPU, ring attention splits the sequence across devices. Re-check long-context quality.

Keep two stores per user: a small profile of stable facts and preferences, always in the prompt, and episode summaries retrieved by relevance.

After each conversation, a model extracts atomic facts with a timestamp and source; newer facts override conflicting ones, and duplicates merge. At query time, fit the profile plus the top retrieved memories into a fixed token budget.

The hard part is trust: users can see and delete memories, sensitive data needs consent and retention limits, stores are isolated per user, and remembered text is data, never instructions. Measure whether it helps.

RAG and fine-tuning fill different gaps: RAG supplies changing or citable facts, fine-tuning shapes format, style or a narrow skill. Over the window: retrieve only the relevant chunks, or summarise in a map-reduce pass. Conversation state: keep recent turns verbatim, summarise older ones, and store durable facts outside the prompt. Prompts: version them like code, with templates and a regression suite. Treat retrieved text as data, never instructions, to limit prompt injection.

def load_history(fetch_page):
    msgs, cursor = [], None
    while True:
        page = fetch_page(cursor)
        msgs.extend(page["messages"])
        cursor = page.get("next_cursor")
        if not cursor:
            return msgs

def compact(msgs, limit, count, summarize, keep=6):
    if sum(count(m) for m in msgs) <= limit:
        return msgs
    while True:
        cut = max(len(msgs) - keep, 0)
        note = {"role": "system",
                "content": summarize(msgs[:cut])}
        out = [note] + msgs[cut:]
        if sum(count(m) for m in out) <= limit:
            return out
        if keep == 0:
            raise ValueError("summary exceeds limit")
        keep //= 2

Follow the cursor until none comes back, retrying failed pages. Compact only when over the limit: summarise older turns into one note, keep the recent turns verbatim, and halve that window if it still doesn't fit. Count with the model's own tokenizer, and never separate a tool call from its result.

Messages are role and content pairs: system, user, assistant. Store each session's list plus a summary by session ID.

Before each call, count tokens: keep the system prompt, a summary of older turns and the latest turns, reserving room for the reply. Stream deltas to the client and save the assembled reply. On 429s and 5xx, retry with jittered exponential backoff, honour Retry-After, and cap attempts.