Interview prep

A list of integer IDs, not the string. A tokenizer swaps each piece of text for an ID from a fixed vocabulary; the model reads IDs and writes them one at a time. With GPT-4o's o200k_base, "Tokenizers turn text into integers." is seven tokens: Token + izers, and the period on its own.

A token is a piece of text from the tokenizer's fixed vocabulary, stored as an integer ID. Subwords are the compromise: whole words need a huge vocabulary and miss new words; characters make sequences several times longer. Rare words are built from frequent fragments, and a byte-level base like GPT-2's encodes any input.

Training repeatedly merges the most frequent adjacent pair of symbols into a new one, giving an ordered merge list. Encoding splits new text into base symbols and replays the merges in order, so an unseen word becomes several known pieces. GPT-2's 50,257 entries are 256 byte tokens, 50,000 merges and one end-of-text token.

Prices are per token of one specific tokenizer, and words-to-tokens isn't constant. The rule of about 100 tokens per 75 words is for English prose with GPT-4's tokenizer. With cl100k_base, an English sentence took 6 tokens and a Hindi one saying roughly the same took 21, so the Hindi tickets would be badly underestimated.

Sample real tickets, count them with the tokenizer of the model you'll call, and budget for output tokens.

Anything that stores tokens, because tokenizer mismatches fail silently: decoding IDs with the wrong tokenizer returns plausible garbage, not an error. The plan:

• Cached token IDs: rebuild them.
• Chunks sized in tokens: re-chunk and re-index.
• Token budgets and cost estimates: recount; the same text gives different counts.
• Throughput dashboards: tokens per second now means a different amount of text, so old and new numbers don't compare.

Keeping the old tokenizer isn't an option: model and tokenizer were trained together.

Block it: that change is the injection. Special tokens like <|endoftext|> carry control meaning, so with allowed_special="all" a user who types the marker can end a turn early or forge a boundary. The ValueError is the defence working.

Encode untrusted text with encode_ordinary() or disallowed_special=(), so the marker stays plain characters; allowed, it becomes ID 199999, a real end-of-text signal. Add special tokens only from your own code. And catching the error to drop the ticket isn't a fix either: it silently loses data.

Word vocabularies are huge and fail on unseen words; characters never fail but make sequences long. Subword methods, BPE, WordPiece and unigram, sit between: frequent words stay whole and rare ones split, so nothing is out of vocabulary.

For the recommender, keep a classic retrieval stage over item and user-history embeddings, and use the LLM to rerank and explain a short candidate list, since item IDs make poor tokens and catalogues change daily.

Train byte-level BPE on a sample of the real mix of languages and code, so nothing is unknown. Vocabulary size trades shorter sequences against bigger embedding and output tables. Add digit and whitespace rules and chat tokens.

KL-regularised SFT adds β times the KL divergence from a frozen base model's next-token distributions to the cross-entropy loss. It limits drift and forgetting; β sets how far it may move.

Subword tokenizers keep common words whole and split rare ones into frequent pieces: no unknown words, short sequences. SentencePiece trains on raw text, spaces included, so it needs no language-specific pre-splitting.

Variants: encoder-only like BERT for understanding, decoder-only like GPT for generation, and encoder-decoder like T5 for input-to-output tasks such as translation.

Work on bytes so any text encodes. Training merges the most frequent adjacent pair into a new ID until its count falls below the threshold.

from collections import Counter

def pairs(ids):
    return zip(ids, ids[1:])

def merge(ids, pair, new):
    out, i = [], 0
    while i < len(ids):
        if tuple(ids[i:i + 2]) == pair:
            out.append(new)
            i += 2
        else:
            out.append(ids[i])
            i += 1
    return out

def train(text, min_count):
    ids, merges = list(text.encode()), {}
    while len(ids) > 1:
        pair, n = Counter(pairs(ids)).most_common(1)[0]
        if n < min_count:
            break
        merges[pair] = 256 + len(merges)
        ids = merge(ids, pair, merges[pair])
    return merges

def encode(text, merges):
    ids = list(text.encode())
    while len(ids) > 1:
        cand = [p for p in set(pairs(ids)) if p in merges]
        if not cand:
            break
        p = min(cand, key=merges.get)  # earliest merge
        ids = merge(ids, p, merges[p])
    return ids

def decode(ids, merges):
    vocab = {i: bytes([i]) for i in range(256)}
    for (a, b), i in merges.items():  # in merge order
        vocab[i] = vocab[a] + vocab[b]
    return b"".join(vocab[i] for i in ids).decode(
        "utf-8", errors="replace")

Encoding applies merges in training order, always the earliest-learned pair present; decoding concatenates byte strings. Interviewers check the round trip, overlapping pairs like aaa, and the stopping rule.

Interface: one keyphrase per text or several, overlapping or not, labelled as offsets or strings?

Tokenizer: which vocabulary, cased or not. Map character labels to token start and end with the fast tokenizer's offset mapping, and decide what happens when a label cuts through a subword. Texts over 512 tokens need sliding windows.

Valid spans: end not before start, a maximum length, a no-answer case. Outputs: top spans as character offsets with scores.

It's a learned matrix with one row per vocabulary entry, and embedding a token just reads its row. all-MiniLM-L6-v2's table is 30,522 × 384, about half its parameters.

No single coordinate means anything; training shapes which vectors land near which. "Similar" means pointing the same way, usually measured with cosine similarity.

Only the last two change with context:

• Static token embedding: a table row, identical in every sentence: river "bank" and money "bank" score cosine exactly 1.
• Contextual token embedding: the transformer's output at one position, so it depends on context.
• Text embedding: one vector per text; MiniLM averages its token outputs, then normalizes.

Normalize every vector, documents and queries alike, to length 1. Cosine is the dot product divided by both lengths, so on unit vectors the two are the same number.

Without that, the dot product also rewards length, so a long document can outrank a closer short one. Verify rather than assume: for the lesson's sentences, MiniLM's vectors were 4.86 to 6.46 long until its final normalize step.

Three risks. First, similarity tracks topic, not truth: in our MiniLM data, "This medicine is safe for children." and its negation scored 0.936, far above the best real paraphrase pair at 0.645. For medicine, negation needs explicit tests.

Second, 0.5 means nothing universal. Scores aren't calibrated and there's always a nearest neighbour, so pick the threshold from labelled real queries and recheck it when the model changes.

Third, MiniLM truncates after 256 word pieces according to its model card, so chunk long leaflets.

Each model arranges its own space, so matching dimensions only means the dot product can be computed. New-model queries against old-model documents give effectively random scores, with no error.

Migrate by re-embedding the whole collection, cutting queries and documents over together, and re-checking every threshold, since thresholds are model-specific. Even a preprocessing change invalidates stored vectors. Budget for it: generating embeddings costs money, and storage grows with dimension. At 4-byte floats, a million documents is about 1.5 GB at 384 dimensions against 12.3 GB at 3,072.

That's the vector's dimension: each text becomes a point with that many coordinates. More dimensions can hold finer distinctions, but every vector costs more memory and search time. The numbers mean nothing alone: vectors compare only with others from the same model.

By vocabulary, each device owns a range of rows, looks up the tokens it owns, writes zeros for the rest, and an all-reduce sum assembles full vectors. It pairs with a tied output head sharded the same way, but popular tokens can make one shard hot.

By hidden dimension, every device holds all rows but a slice of the width. Lookups are local, then an all-gather concatenates the slices. Both move about batch × tokens × width numbers; choose by what the next layer wants.

Embeddings: shortlist models on public retrieval benchmarks, then pick on your own queries; smaller dimensions or int8 vectors cut memory and latency.

Index: HNSW gives high recall at low latency but holds full vectors in memory; IVF with product quantisation compresses billions of vectors. Tune the recall-latency trade with ef or nprobe, then rerank a shortlist exactly.

Sharding: partition by user or tenant, since an assistant searches one person's content, which also enforces isolation; otherwise scatter-gather across shards and merge top-k. Add replicas for throughput.

Two endpoints, one encoder: classification is a light head on the embedding. Interactive calls are synchronous, with dynamic batching to a latency target.

Batch is an async job: submit a file, get a job id, poll or take a webhook, read results from storage, trading latency for cost. Every response names the model version, because vectors from different versions aren't comparable.

Build a collaboration graph: edges from shared documents, meetings, messages, code reviews and the org chart, weighted by frequency and recency. Learn node embeddings, with node2vec or a graph neural network, combine them with text embeddings of each person's work, and use cosine distance.

Validate by predicting held-out future collaborations, then by people-search clicks. Respect permissions: only use signals the viewer could see.

Yes: a two-tower model with user and item embeddings, trained with in-batch negatives. A positive should mean value: a click with real dwell time or a save, never a bare impression.

Retrain on a time-based split, computing features only from data before each label's timestamp, so nothing leaks from the future. Exposure bias: the logs only hold what the old model chose to show, so keep a small exploration slice of randomised items, correct for popularity in sampled negatives, and weight by inverse propensity. Validate on the following day, then A/B test.

In high dimensions data gets sparse and distances concentrate, so models need far more data. PCA finds the linear projection that keeps the most variance: fast, deterministic, and easy to size from explained variance. An autoencoder learns a nonlinear encoder and decoder by minimising reconstruction error. A linear autoencoder with squared loss spans the same subspace as PCA, so it only wins on nonlinear structure with enough data. Judge them on the downstream task, not reconstruction: train the same model on each representation, sweep the dimension, and compare held-out accuracy and stability across seeds.

import numpy as np

def top_k_cosine(q, docs, k=5, eps=1e-12):
    n = np.linalg.norm(docs, axis=1, keepdims=True)
    d = docs / np.maximum(n, eps)
    qn = q / max(np.linalg.norm(q), eps)
    scores = d @ qn
    idx = np.argsort(-scores)[:k]
    return idx, scores[idx]

Cosine is the dot product of unit-length vectors, so normalise once at indexing and any dot-product index ranks by cosine. Raw dot product also rewards vector length. Follow how the embedding model was trained: cosine for most sentence-embedding models, dot product when the model puts meaningful signal, such as popularity, in the norm. Guard against zero vectors.

A two-tower model. The item tower embeds image and text content, so new items get vectors immediately. The user tower runs a transformer over recent actions plus pooled summaries of older history, since 160,000 events won't fit attention.

Build each example as of its timestamp: only earlier events, split train and test by time. For freshness, refresh user embeddings daily and add recent actions at request time. New users fall back to onboarding interests.

Join each impression to the profile as of that timestamp, label clicks positive and skips negative, then split by time: train on the past, validate on the next window, and keep anything fitted, like vocabularies or normalizers, to the training slice.

Model: a two-tower text encoder for member and item, trained contrastively with in-batch negatives plus skips as hard negatives. Evaluate recall at k against a popularity baseline.

For reproducibility, pin seeds, model versions and a data-snapshot hash in one config, ship preprocessing with the model, and test that inference reproduces the training embeddings.

Uploads land in object storage and enqueue a job. Workers per modality: documents are parsed and chunked by structure; images are embedded directly; videos are sampled at intervals and scene changes, with the audio transcribed and embedded as text. Batch on GPUs.

Idempotency: key each output by content hash, model version and chunking config, and upsert, so retries never duplicate. Failures go to a dead-letter queue.

Store vectors with tenant and offsets. Tenant isolation is enforced server-side: per-tenant namespaces, per-tenant keys, and deletes that propagate to vectors. Model upgrades re-embed into a new index.

Text arrives on a log. Stream workers tokenize with the tokenizer pinned to the encoder version, embed in GPU micro-batches, and store vectors with ID, timestamp and version.

Train a light head, logistic regression or an MLP, on frozen embeddings, joining labels by time; retrain periodically and promote only after validating on the newest window.

Serving latency is mostly the encoder: use a small distilled one, batch, and cache repeated texts. Keep training and serving on one embedding version, and backfill when it changes.

Score each token's query against every visible key, divide by √d, softmax, and weight the values. The query is what a token looks for, the key what it offers, the value what it hands over. Dot products grow with d, and large scores push softmax onto one token, where gradients get tiny.

Two sub-blocks, each layer norm, then the sublayer, then a residual add: first multi-head attention, then an MLP.

Attention moves information between tokens; the MLP transforms each token on its own and holds most parameters. In GPT-2 small's first block, the MLP has 4,722,432 against attention's 2,362,368, and the model stacks 12 blocks.

It hides later tokens: before softmax, every score whose key comes after the query becomes minus infinity, so those weights are exactly 0 and the rest still sum to 1.

So a decoder disambiguates only from the past: whether the ladder sentence ends "too short" or "too high", GPT-2 small's attention rows up to and including " it" were identical in all 12 layers and 12 heads. That also makes the KV cache exact.

It's a lead to test, not a conclusion. Head 4.3 puts 0.659 on " ladder" from " it", but " did", " not" and " reach" also look mostly at " ladder", and the subject always sat in the first two tokens. That can't separate "finds the subject" from "likes an early position".

Attention weights show where a head reads from, not what the values carry or how 144 heads combine. I'd test sentences with the noun in different positions.

Attention compute grows quadratically. Each head scores an n × n grid, so doubling n roughly quadruples that work; the paper puts self-attention per layer at n² · d. GPT-2's 1,024-token window is about a million scores per head. The KV cache grows linearly, with context and batch size, and with learned positions the window is a hard cap.

Fixes trade flexibility for speed and memory: a local attention window, keys and values shared across heads, a cache in fewer bits, or architectures that scale linearly. I'd measure latency and memory at both lengths first.

Project once, split into heads, attend in each head, merge.

def mhsa(x, w_qkv, w_o, h):            # x: (B, T, d)
    B, T, d = x.shape
    q, k, v = (x @ w_qkv).split(d, -1)
    heads = lambda t: t.view(B, T, h, -1).transpose(1, 2)
    q, k, v = heads(q), heads(k), heads(v)  # (B, h, T, d/h)
    s = q @ k.transpose(-2, -1) / (d // h) ** 0.5
    future = torch.ones(T, T, dtype=torch.bool,
                        device=x.device).triu(1)
    s = s.masked_fill(future, float("-inf"))
    y = s.softmax(-1) @ v
    return y.transpose(1, 2).reshape(B, T, d) @ w_o

Interviewers watch for the mask before softmax, scaling by the head size, not d, and the transpose before the final reshape.

Pre-norm should read x + attn(norm(x)), then x + mlp(norm(x)): normalise each sublayer's input and add its output to the un-normalised stream.

Common bugs: adding to the normed tensor, so the residual path stops being an identity; normalising the sublayer's output instead of its input; one norm shared where two are meant; and no final norm before the output layer. A quick test: zero both output projections, and the block must return x exactly.

LayerNorm subtracts the mean, divides by the standard deviation, then applies a learned scale and shift. RMSNorm skips the mean: it divides by the root mean square and keeps only the scale. That's less computation with comparable quality in the paper's experiments, so models like Llama use it, in the same pre-norm positions.

def sdpa(q, k, v, mask=None):  # mask True = may attend
    s = q @ k.transpose(-2, -1) / q.size(-1) ** 0.5
    if mask is not None:
        s = s.masked_fill(~mask, float("-inf"))
    s = s - s.amax(-1, keepdim=True)
    p = s.exp()
    return (p / p.sum(-1, keepdim=True)) @ v

Scale by √d so scores don't saturate softmax. Subtract the row max before exp so nothing overflows; torch's softmax does this internally. A fully masked row gives NaN, since max minus max is inf minus inf, so guard it. In FP16, compute scores and softmax in FP32. FlashAttention does the same blockwise with a running max.

def self_attn(x, wq, wk, wv):        # x: (T, d)
    q, k, v = x @ wq, x @ wk, x @ wv
    s = q @ k.T / q.shape[-1] ** 0.5  # (T, T)
    s = s - s.max(-1, keepdims=True)
    p = np.exp(s)
    return (p / p.sum(-1, keepdims=True)) @ v

Time is O(T² d) and the score matrix takes O(T²) memory, against O(T d²) for projections, so long sequences are attention-dominated. FlashAttention stays exact but tiles the work so the T-by-T matrix never reaches HBM. Sliding-window and sparse patterns make it linear in T; linear and low-rank attention approximate; MQA and GQA shrink the KV cache.

import math

def sigmoid(x):
    if x >= 0:
        return 1 / (1 + math.exp(-x))
    e = math.exp(x)  # x is negative: no overflow
    return e / (1 + e)

def softmax(xs):
    m = max(xs)  # shift so the largest exp is 1
    es = [math.exp(x - m) for x in xs]
    s = sum(es)
    return [e / s for e in es]

def attention(Q, K, V):
    d = len(Q[0])
    out = []
    for q in Q:
        scores = []
        for k in K:
            dot = 0.0
            for a, b in zip(q, k):
                dot += a * b
            scores.append(dot / math.sqrt(d))
        w = softmax(scores)
        row = [0.0] * len(V[0])
        for wi, v in zip(w, V):
            for j in range(len(v)):
                row[j] += wi * v[j]
        out.append(row)
    return out

The trap is overflow: exp of a large number blows up. Branch sigmoid on the sign, and subtract the max before softmax, which leaves the result unchanged. Attention is then dot products, scaled by √d, a softmax per query, and a weighted sum of values.

Self-attention: each token scores its query against every key, softmaxes, and sums the values by weight.

LoRA: freeze the weights, train a low-rank update.

Adam tracks running averages of each parameter's gradient and squared gradient, giving per-parameter step sizes: the transformer default. SGD uses one global rate and less memory; it often generalises well in vision.

ViT cuts an image into patches, embeds each as a token, and runs a standard transformer encoder.

import torch, torch.nn as nn, torch.nn.functional as F

class Block(nn.Module):
    def __init__(self, d, h):
        super().__init__()
        self.h = h
        self.ln1 = nn.LayerNorm(d)
        self.ln2 = nn.LayerNorm(d)
        self.qkv = nn.Linear(d, 3 * d)
        self.proj = nn.Linear(d, d)
        self.mlp = nn.Sequential(
            nn.Linear(d, 4 * d), nn.GELU(),
            nn.Linear(4 * d, d))

    def attn(self, x):
        B, T, C = x.shape
        q, k, v = self.qkv(x).split(C, dim=-1)
        q, k, v = (t.view(B, T, self.h, -1)
                   .transpose(1, 2) for t in (q, k, v))
        s = q @ k.transpose(-2, -1) / k.shape[-1] ** 0.5
        m = torch.ones(T, T, dtype=torch.bool,
                       device=x.device).tril()
        s = s.masked_fill(~m, float("-inf"))
        y = s.softmax(-1) @ v
        y = y.transpose(1, 2).reshape(B, T, C)
        return self.proj(y)

    def forward(self, x):
        x = x + self.attn(self.ln1(x))
        return x + self.mlp(self.ln2(x))

class Decoder(nn.Module):
    def __init__(self, vocab, d, h, n, max_len):
        super().__init__()
        self.tok = nn.Embedding(vocab, d)
        self.pos = nn.Embedding(max_len, d)
        self.blocks = nn.ModuleList(
            Block(d, h) for _ in range(n))
        self.ln = nn.LayerNorm(d)
        self.head = nn.Linear(d, vocab, bias=False)

    def forward(self, idx, targets=None):
        pos = torch.arange(idx.shape[1],
                           device=idx.device)
        x = self.tok(idx) + self.pos(pos)
        for b in self.blocks:
            x = b(x)
        logits = self.head(self.ln(x))
        if targets is None:
            return logits
        loss = F.cross_entropy(logits.flatten(0, 1),
                               targets.flatten())
        return logits, loss

Interviewers watch the causal mask, head reshapes that keep batch and heads apart, pre-norm residuals, and targets shifted by one token outside the model.

The candidate item is the query; the user's behaviour sequence supplies the keys and values. Each past item is embedded from its ID, category and context like time gap, then projected into a key and a value; the candidate's embedding is projected to the query.

Softmax over query-key scores weights the history, so the output is a candidate-specific interest vector, the target attention DIN introduced. Cost grows with history length per candidate, so long histories are truncated or filtered first.

State shapes as you go: x is T by d, scores T by T. Mask before the softmax, and normalise stably.

def attention(x, Wq, Wk, Wv, mask):   # x: (T, d)
    q, k, v = x @ Wq, x @ Wk, x @ Wv  # (T, d_k), (T, d_v)
    s = q @ k.T / np.sqrt(k.shape[-1])  # (T, T)
    s = np.where(mask, s, -1e9)       # mask: True = keep
    s = s - s.max(-1, keepdims=True)
    p = np.exp(s)
    p /= p.sum(-1, keepdims=True)
    return p @ v                      # (T, d_v)

def cross_entropy(logits, y, ignore=-100):  # (N, V), (N,)
    keep = y != ignore
    z, t = logits[keep], y[keep]
    m = z.max(-1, keepdims=True)
    lse = m[:, 0] + np.log(np.exp(z - m).sum(-1))
    return (lse - z[np.arange(len(t)), t]).mean()

Cross-entropy is log-sum-exp minus the target logit, averaged over non-ignored tokens. Interviewers check the scale by the key width, masking scores rather than zeroing probabilities, and that padding labels are skipped.

RNNs carry one hidden state step by step: cheap per step, but sequential, and gradients vanish over long spans. LSTMs add gates and a cell state so information survives longer. Transformers attend over all steps in parallel, train faster and capture long range, at cost quadratic in length; for streaming prediction I'd pick a causal decoder.

MPC is different: it uses a dynamics model to optimise actions over a short horizon, applies the first and replans. A learned sequence model can be that dynamics model or forecaster, while MPC enforces constraints.

Have the forward pass cache what the backward pass needs: the softmax probabilities P.

def attn_fwd(Q, K, V):              # (T, d) each
    S = Q @ K.T / np.sqrt(Q.shape[-1])
    S = S - S.max(-1, keepdims=True)
    P = np.exp(S)
    P /= P.sum(-1, keepdims=True)
    return P @ V, P

def attn_bwd(dO, Q, K, V, P):
    dV = P.T @ dO
    dP = dO @ V.T
    dS = P * (dP - (dP * P).sum(-1, keepdims=True))
    dS /= np.sqrt(Q.shape[-1])
    return dS @ K, dS.T @ Q, dV     # dQ, dK, dV

The key line is the softmax backward: each row's gradient is P times dP minus its P-weighted mean. Multi-head attention reshapes into heads around this, and a block adds residuals, LayerNorm and an MLP, each with its own backward. Prove it with a finite-difference gradient check.

Convolutions build in locality and translation equivariance: each layer sees a small neighbourhood so they learn well from less data, but long-range links need depth or pooling. Self-attention connects every position to every other in one layer with few built-in assumptions, so it needs more data or pretraining, and its cost grows quadratically with sequence length. Hybrids take both: a convolutional stem feeding transformer layers, or convolution stages early and attention later, as in CoAtNet.

Masks set scores to minus infinity before the softmax. The causal mask hides future positions; the padding mask hides pad keys, and pads are excluded from the loss.

In cross-attention, queries come from the decoder, keys and values from the encoder.

At decode time, past keys and values never change, so the KV cache stores them; each step computes projections for one new token instead of re-running the prefix.

Mixed precision runs matmuls in FP16 or BF16, keeps FP32 master weights and optimizer state, and with FP16 scales the loss so small gradients don't underflow.

import math
import torch

def sinusoidal_pe(T, d):
    pos = torch.arange(T).float()[:, None]   # (T, 1)
    i = torch.arange(0, d, 2).float()        # (d/2,)
    freq = torch.exp(-math.log(10000.0) * i / d)
    pe = torch.zeros(T, d)                   # d even
    pe[:, 0::2] = torch.sin(pos * freq)
    pe[:, 1::2] = torch.cos(pos * freq)
    return pe

# x: (B, T, d); x = x + sinusoidal_pe(T, d).to(x)
# learned: x = x + nn.Embedding(max_len, d)(pos_ids)

Dimension pair 2i gets sin and cos of pos divided by 10000 to the power 2i/d. A learned table is simpler but can't go past max_len.

Without positions, attention is permutation-equivariant: an encoder sees a bag of words, so shuffled inputs give shuffled outputs and order-dependent tasks fail. A causal decoder recovers some position from its mask.

Logits are the raw scores the LM head gives every vocabulary token, 50,257 for GPT-2, from the last position's vector. Softmax turns them into probabilities: after "Thank you very", " much" got 0.992.

For a sentence, loop: pick a token, append it, run the model again. So 25 new tokens means 25 passes.

Temperature divides every logit by T before softmax: low T sharpens, high T flattens, and T near 0 approaches greedy. Top-k keeps a fixed number of top tokens; top-p keeps the smallest set whose probabilities reach p, so the cut adapts to the distribution's shape.

At top-p 0.9, GPT-2 kept 1 token after "Thank you very" but 417 for an open-ended prompt. No single k suits both: Holtzman et al.'s argument.

It's a feedback loop: once a phrase has appeared, the model rates it more likely to appear again, and greedy always takes the top choice. Continuing "I opened the box and found", GPT-2 small picked " amount" at 0.027 the first time and 0.988 on the repeat.

Holtzman et al. report the same for most phrases they tested: decoding that maximizes probability tends to give bland or looping text.

The test assumes a determinism a hosted API doesn't promise. Temperature 0 is greedy and uses no random numbers, and our sampling script gave byte-identical output across two runs, but on one machine. AI Engineering notes that even with temperature, top-p, top-k and seed fixed, hardware can change the output, and with an API the hardware is the provider's choice.

So stop asserting exact text from live calls: cache known answers where you need exact strings, or parse the output and compare fields, labels or structure. A seed won't fix it.

Against it: a token probability says how typical the text is after the prompt, not whether it's correct. Asked for the largest planet, GPT-2 small rated " Earth" at 0.0282, slightly above " Jupiter" at 0.0270. The number also moves with sampling settings, and the GPT-4 report found the model much less calibrated after post-training.

Instead, use logprobs internally: log them, flag low-probability answers for review, and check on a labelled set that the flag actually catches errors.

Keep the k best partial sentences. Each step, extend every one with its likely next tokens, score by the sum of log-probabilities, and keep the top k. A sentence's probability is the exponent of that sum.

Greedy is beam width 1, so it misses sequences whose first token wasn't the best. Longer sentences collect more negative terms, so normalise by length when comparing finished beams.

Count each word's successors, then sample in proportion to the counts.

import random
from collections import Counter, defaultdict

def train(words):
    counts = defaultdict(Counter)
    for a, b in zip(words, words[1:]):
        counts[a][b] += 1
    return counts

def next_word(counts, w):
    nxt = counts.get(w)
    if not nxt:
        return None              # unseen word
    ws, cs = zip(*nxt.items())
    return random.choices(ws, weights=cs)[0]

Interviewers check the unseen-word case and whether you can sample without random.choices: a cumulative sum plus binary search.

Self-attention draws queries, keys and values from one sequence; cross-attention takes queries from the decoder and keys and values from another source, like an encoder's output.

Decoding is a loop: score the next token, pick one greedily or by temperature and top-p, append, repeat, with a KV cache. RL then tunes those sampled outputs against a reward, held near a reference model. Evaluate with perplexity, task benchmarks and human or judge preferences.

Greedy takes the argmax. Both samplers sort the probabilities, keep a prefix, renormalise it and sample.

import numpy as np

def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()

def pick(logits, rng, mode="greedy", k=5, p=0.9,
         temp=1.0):
    if mode == "greedy":
        return int(np.argmax(logits))
    probs = softmax(logits / temp)
    order = np.argsort(probs)[::-1]   # high to low
    if mode == "top_k":
        keep = order[:k]
    else:                             # top_p
        cum = np.cumsum(probs[order])
        keep = order[:np.searchsorted(cum, p) + 1]
    q = probs[keep] / probs[keep].sum()
    return int(rng.choice(keep, p=q))

Interviewers check the max subtracted in softmax, that top-p keeps the token that crosses p, and the renormalisation before sampling.

It's the most tokens a model can read at once, shared by prompt, history and output: 1,024 for GPT-2 small. Attention alone ignores order, so GPT-2 adds a learned position vector from a table with exactly 1,024 rows. Position 1,025 has no row; on our CPU run it raised an IndexError.

The wait is prefill. Prefill runs the whole prompt in one parallel pass and sets the time to first token; decode then writes one token per step, each costing about the same.

On our test machine, GPT-2 small prefilled 16 tokens in 15.3 ms and 960 in 219.6 ms, while a decode step stayed between 8.0 and 9.6 ms. The lever is how much text goes into the prompt.

It stores every processed token's keys and values, per layer and head, so each decode step computes only the new token's.

It's exact because of the causal mask: earlier tokens never depend on later ones, and greedy output was identical with and without the cache. After a 960-token prompt on our test machine, a step took 9.6 ms with the cache against 228.8 ms without, about 24 times faster.

Multiply it out: 2 (key and value) × layers × heads × head dimension × tokens × bytes per number × batch. GPT-2 small in 32-bit floats needs 72 KiB per token; a full 1,024-token window at batch 64 is 4.50 GiB, nearly ten times the weights. Cache, not weights, can cap context length and concurrent users.

Each factor is a lever: fewer bytes (a lower-precision cache), fewer tokens (a local window, trimmed history), keys and values shared across heads, a smaller batch. I'd model the real mix of context lengths and batch sizes before buying hardware.

A. On our GPT-2 small measurements, A is about 220 ms of prefill plus 10 steps of about 10 ms, roughly 0.3 s; B is about 15 ms plus 200 sequential steps of about 8 ms, roughly 1.6 to 1.7 s. Output tokens run one after another, while prompt tokens are processed together.

Resending the history re-prefills more every turn, so time to first token and cache memory grow turn by turn; once history won't fit the window, apps drop or summarise old turns. APIs typically bill input and output tokens, so trim both.

In multi-head attention every head has its own queries, keys and values. Grouped-query attention lets a group of query heads share one key-value head.

The cache holds keys and values per key-value head, so fewer of them means a proportionally smaller KV cache: Llama 2 70B's 64 query heads share 8, an 8× cut, with little quality loss.

Usual planted bugs: the mask applied after softmax, scaling by d instead of head size, a missing transpose before merging heads. For the cache, each decode step computes q, k and v for the new token only, appends k and v, and attends over everything cached.

def step(x, w_qkv, w_o, h, cache):  # x: (B, 1, d)
    B, _, d = x.shape
    q, k, v = (x @ w_qkv).split(d, -1)
    sh = lambda t: t.view(B, 1, h, -1).transpose(1, 2)
    q, k, v = sh(q), sh(k), sh(v)   # (B, h, 1, d/h)
    if "k" in cache:
        k = torch.cat([cache["k"], k], dim=2)
        v = torch.cat([cache["v"], v], dim=2)
    cache["k"], cache["v"] = k, v
    s = q @ k.transpose(-2, -1) / (d // h) ** 0.5
    y = s.softmax(-1) @ v           # no mask: all past
    return y.transpose(1, 2).reshape(B, 1, d) @ w_o

Interviewers check that cached decoding matches the full forward pass and that positions continue from the cache length.

Write it once as GQA: plain MHA is the case where key-value heads equal query heads, and multi-query is one key-value head.

import torch, torch.nn as nn, torch.nn.functional as F

class GQA(nn.Module):
    def __init__(self, d, n_q, n_kv):
        super().__init__()
        self.hd, self.nq, self.nkv = d // n_q, n_q, n_kv
        self.q = nn.Linear(d, d, bias=False)
        self.kv = nn.Linear(d, 2 * n_kv * self.hd,
                            bias=False)
        self.o = nn.Linear(d, d, bias=False)

    def forward(self, x, cache=None):
        B, T, _ = x.shape
        q = self.q(x).view(B, T, self.nq, self.hd)
        k, v = self.kv(x).view(
            B, T, 2, self.nkv, self.hd).unbind(2)
        q, k, v = (t.transpose(1, 2) for t in (q, k, v))
        if cache is not None:
            k = torch.cat([cache[0], k], dim=2)
            v = torch.cat([cache[1], v], dim=2)
        cache = (k, v)  # n_kv heads: the memory saving
        r = self.nq // self.nkv
        k = k.repeat_interleave(r, dim=1)
        v = v.repeat_interleave(r, dim=1)
        S = k.shape[2]  # cached + new positions
        # new query i sits at position S - T + i
        m = torch.ones(T, S, dtype=torch.bool,
                       device=x.device).tril(S - T)
        a = q @ k.transpose(-2, -1) / self.hd ** 0.5
        a = a.masked_fill(~m, float("-inf"))
        y = F.softmax(a, dim=-1) @ v
        y = y.transpose(1, 2).reshape(B, T, -1)
        return self.o(y), cache

Interviewers check three things: the mask is offset by the cached length, the cache stores keys and values before repeating them, so it shrinks by the group factor, and cached decoding reproduces a full forward pass token for token.

FlashAttention computes exact attention in tiles kept in on-chip SRAM, with an online softmax, so the full score matrix never goes to HBM. Memory becomes linear in length, and it's faster because attention is memory-bound.

The KV cache keeps past keys and values so each decode step computes only the new token, trading memory for compute. RoPE rotates queries and keys by position-dependent angles, so dot products depend on relative position, parameter-free.

Attention builds each token's output as a weighted mix of other tokens. Each hidden vector is multiplied by three learned matrices to give a query, key and value. Scores are query-key dot products divided by the root of the head size, masked causally in a decoder, softmaxed, then used to average the values.

In decoding, past tokens' keys and values never change, so the KV cache stores them. Each step projects only the new token and attends over the cache, instead of reprocessing the prefix, paying in memory that grows with batch times context.

class KVCache:
    def __init__(self, L, B, H, T_max, dh, dtype, dev):
        shape = (L, B, H, T_max, dh)
        self.k = torch.zeros(shape, dtype=dtype, device=dev)
        self.v = torch.zeros_like(self.k)

    def update(self, layer, k, v, start):
        # k, v: (B, H, n, dh); n = prompt len, then 1
        end = start + k.size(2)
        self.k[layer, :, :, start:end] = k
        self.v[layer, :, :, start:end] = v
        return (self.k[layer, :, :, :end],
                self.v[layer, :, :, :end])

    def reorder(self, beam_idx):
        # beam search: rows follow surviving beams
        self.k = self.k.index_select(1, beam_idx)
        self.v = self.v.index_select(1, beam_idx)

Prefill writes the whole prompt in one pass; each decode step writes one position and attends over everything so far. Batch with left padding plus an attention mask so positions line up.

Memory per token is 2 × layers × KV heads × head dim × bytes, so cap batch times length. Beyond that: GQA, quantized caches, or paged blocks that avoid reserving T_max per sequence.

It's fluent, confident output that is false or not grounded in any source. The model was trained to predict the token that actually came next in real text, not the true one, and nothing checks truth. So a wrong answer reads as smoothly as a right one.

Because likely text isn't true text. In our run, " Sydney" was the top token at 0.216 and " Canberra" only fifth at 0.0436, and greedy stated Sydney with no hedge.

There's no lookup: knowledge is spread through the weights, and a plausible wrong answer, the best-known city, can outweigh the right one. We didn't measure why Sydney wins; one plausible reason is that it appears near "Australia" in far more web text.

Every generated token becomes context for the next, and nothing marks it as the model's own guess, so each step is another chance to go wrong. AI Engineering describes snowballing: the model keeps justifying an early wrong assumption.

A right start doesn't protect the rest: GPT-2's greedy Everest answer began correctly, then put the summit at about 1,000 feet. That's one reason concise answers seem to help.

Treat probability as a signal to validate, not a verdict. On our 14 prompts, four right answers, including " Tokyo" at 0.1186, were less probable than the wrong " Sydney" at 0.216, and the GPT-4 report found post-training weakened calibration.

So log the probabilities, flag low ones for review, and measure on labelled traffic how many real errors the flag catches and how many good answers it wrongly flags. Recheck after any model or prompt change. For free-form answers, Kadavath et al. ask the model whether its answer is true and use that probability.

Ground answers in retrieved policy text. Context steers strongly: in our toy, one true sentence moved " Canberra" from 0.0436 to 0.8248, though that mostly measures copying, and retrieval can find the wrong text.

So: retrieve, require a cited passage for each answer and check it supports the claim, keep answers short and restricted to the given context, and let the bot say it doesn't know. Evaluate local factual consistency: is each answer supported by the documents it was given? That's far easier to check than truth against world knowledge.

Make abstaining a valid output. Questions needing fresh or private facts go to retrieval or a tool; if nothing clears a relevance threshold, say so rather than guess. Then check answers: require citations to retrieved passages, and treat disagreement across several samples as an uncertainty signal. Low confidence falls back to a clarifying question or a human. Evaluate on unanswerable questions as well as answerable ones.

Quality is task-specific: reference answers where they exist, a rubric judge, human spot checks. Hallucination against supplied context is faithfulness: split the answer into claims and check each is supported by the context.

import re
from dataclasses import dataclass, field

@dataclass
class Result:
    score: float
    unsupported: list = field(default_factory=list)

def claims(answer):
    parts = re.split(r"(?<=[.!?])\s+", answer)
    return [p.strip() for p in parts if p.strip()]

def faithfulness(answer, context, supports):
    # supports(context, claim) -> bool: an NLI
    # model or a judge prompt
    cs = claims(answer)
    if not cs:
        return Result(1.0)
    bad = [c for c in cs if not supports(context, c)]
    return Result(1 - len(bad) / len(cs), bad)

Interviewers look for claim-level scoring, a pluggable judge validated against human labels, and the empty case handled.

Voice adds two problems: errors start at speech recognition, and the user can't see sources. So use the recogniser's confidence, and confirm an uncertain name or number before acting on it.

Then answer from tools and retrieval, like calendar, contacts and search, not model memory, telling the model to answer only from those results. Check the draft against them before speaking, and confirm before any action. Keep replies short, name the source aloud, and say when it doesn't know. Evaluate on recorded, noisy, accented queries, scoring faithfulness and task success.

The model doesn't know today's date, so tell it: put the date and the training cutoff in the system prompt.

Then train the behaviour. Fine-tune on questions about things that change, like recent events or latest versions, with answers that give what held at the cutoff and flag it may be stale, or call search. Include stable questions so it doesn't over-refuse. Evaluate on dated questions either side of the cutoff.

Not in the control loop. Control runs many times a second and must be verifiable; an LLM's latency varies, and a hallucinated object or rule has no safe failure mode.

Use them off the hot path: auto-labelling and mining rare scenarios from fleet logs, generating simulation scenarios, explaining decisions, and as a slow planner suggesting high-level intent that a fast, verified planner accepts or rejects, as DriveVLM proposes. Anything a VLM contributes online sits behind a deterministic safety layer and is validated in simulation first.

Use vetted clinical guidelines and de-identified, consented conversations, split by patient so nothing leaks into the test set. Start with RAG over trusted guidelines for current, cited facts; fine-tune only for tone or format. Clinicians check each claim against sources, and a test set of unsafe-advice cases, like emergencies, must trigger referral.

Time to first token (TTFT), mostly prefill, and total latency, mostly decode. Streaming makes TTFT the visible wait: on our test laptop's CPU, GPT-2 small's first token arrived at 67.5 ms, the last at 633 ms. A fast reader takes about 120 ms per token, so tokens needn't arrive much faster than that.

Record one timestamp per token, measured from sending (t0). TTFT is t1 − t0; each gap between tokens is inter-token latency; TPOT is their average; total latency is tN − t0, about TTFT + TPOT × (N − 1).

Keep every timestamp: our CPU stream's median gap was 8.6 ms, the slowest 11.5 ms.

Latency is a distribution, and the mean hides the tail. The same request sent 200 times on our laptop CPU had a p50 of 61.3 ms, a p99 of 112.8 ms and a slowest run of 176.2 ms. Users meet the tail: send 100 messages a day and you hit the p99 about daily.

GPU calls are asynchronous: they return before the GPU finishes, so I timed the queuing, not the work. Wait for a result, or synchronise the device, before stopping the clock.

Two more traps give no error: skipping warm-up, since first calls compile code and fill caches; and where the clock sits, since server-side timing leaves out the network and queue users feel.

TTFT counts the first generated token; the user waits for the first visible one, after every hidden plan or reasoning token, plus any tool calls. Some teams call that time to publish. In our CPU stream with the first 48 tokens hidden, the first visible token came at 504.2 ms instead of 67.5 ms. Measure what the user sees, alongside TTFT.

Batching raises total throughput and slows each user. On our laptop CPU, 16 requests decoded together gave 863 tokens/s in total against 125 for one, but each user dropped to 54.

So I'd decide on goodput: requests per minute meeting the SLO, say first token within 100 ms and at most 10 ms per token. Throughput also counts requests users gave up on, so goodput can fall while it rises. On a 120-request CPU workload, 83 met that SLO.

A decoder-only model runs prefill, then one decode step per token, and decode is memory-bound, so the win is batching many requests.

Continuous batching adds and removes requests at every step instead of waiting for a whole batch to finish. PagedAttention stores each request's KV cache in fixed-size blocks, like memory pages, so nothing is reserved for the maximum length and wasted: more requests fit, and shared prefixes can share blocks.

First measure which wait it is: time to first token or total time, at p50 and p99.

Slow first token: shorten the prompt, cache the static prefix, cut queueing. Slow total: cap output length, use a smaller or quantized model, check batching. Always stream, so users read while it writes, and fix the biggest measured part first.

Batch at the iteration level: with continuous batching, finished sequences leave and queued ones join every decode step, so short requests never wait for long ones. The scheduler admits work while KV-cache memory allows, using paged allocation, and chunks long prefills so they don't stall decodes.

Route by queue depth and free cache, preferring a replica that holds the prompt prefix. Autoscale on queue length and time to first token, not GPU utilisation, keep warm spares because loading weights is slow, and shed batch traffic first under overload.

If a batch costs 100 ms whatever its size, throughput is batch size divided by 0.1 s, so bigger batches are free until memory runs out. Latency is the wait for the next batch plus 100 ms. A batcher fires when the batch fills or a timeout expires. Bound the queue and reject when full, so latency stays predictable. For long generations, batch per decode step instead.

Three jobs. Admission: per-tenant quotas counted in tokens, not requests, with token buckets and priority classes so interactive traffic beats batch. Fair scheduling inside each replica: weighted fair queuing across tenants, so one heavy tenant can't fill every batch slot. Routing: choose replicas by load and prefix-cache affinity, so repeated system prompts hit warm KV cache, spilling over when a replica runs hot.

Track time to first token and per-token latency per tenant against SLOs, and under overload shed the lowest priority first with retry-after errors.

Profile first, then attack the binding limit. Training: BF16 or FP8 mixed precision, FlashAttention and fused kernels, activation checkpointing to trade compute for memory, ZeRO or FSDP sharding, and parallelism laid out to the topology, tensor parallel inside NVLink, pipeline and data across nodes, with communication overlapped. Track MFU.

Inference: decode is memory-bound, so cut bytes with quantised weights and KV cache and GQA, raise batch size with paged KV and continuous batching, use speculative decoding for latency, and add tensor parallelism only as far as latency needs.

Assume 100 million users each send 10 requests a day: about 11,600 QPS average, peaks perhaps 2 to 3 times higher. Weights are parameters times bytes, so a 70B model in 16-bit is 140 GB, sharded across GPUs. KV cache per token is 2 times layers times KV heads times head dimension times bytes, times context and concurrency.

For tail latency: continuous batching, admission control, split prefill and decode, autoscale on queue depth.

Assume the provider will be slow, down or wrong. Set timeouts on time to first token, retry with backoff and jitter only on safe errors, and use a circuit breaker to fail over to another provider or smaller model. Validate each output against a schema and policy checks before showing it; on failure, retry once, then use a scripted reply or human handoff. Keep state server-side so failover resumes cleanly.

One API; adapters normalize each provider's schema and streaming. Route by model alias, cost and region; fall back only to models evaluated as equivalent.

Quotas: token buckets per tenant, counting requests and tokens. Retry only timeouts, 429s and 5xx, with jittered backoff, honouring Retry-After, under a retry budget. A circuit breaker per provider opens on error rate, then sends probes.

Log metadata always: tokens, latency, cost, status. Bodies are redacted or skipped, with short retention.

Profile first. Cut padding with dynamic padding and length bucketing, batch requests, and serve through an optimized runtime with fused kernels in FP16 or INT8. If that's not enough, distill into a smaller model.

Check quality on a held-out set: accuracy and F1 overall and per slice, plus agreement with the original. Then shadow traffic.

A router consults a model registry that maps each model name to versions, endpoints and traffic splits, enabling canary, shadow and A/B.

Artifacts are immutable and versioned together: weights, preprocessing and feature schema, with lineage to data and code. CI checks offline metrics, schema compatibility and latency before promotion; canaries roll back automatically on SLO breaches.

Features come from an online store by entity key, with timeouts and defaults, using the same transforms as training; log what was served. Batch dynamically and autoscale on queue depth.

Compute-bound work is limited by arithmetic; bandwidth-bound work by how fast data moves from memory. Arithmetic intensity, operations per byte read, decides which. Decode is bandwidth-bound: each step reads every weight for one token.

GPT-2 small in 4-byte floats reads 497.8 MB per token, about half an operation per byte.

Measure both ceilings, a large memory copy and a large matrix multiply, and place each phase on a roofline. On our test laptop's CPU, a batch-1 decode step moved 507 MB in 9.1 ms: 55.7 GB/s, 46% of the copy ceiling, at about 0.5 operations per byte, far left of the ridge at 16. A 960-token prefill reached 51% of the matrix-multiply ceiling. For decode, more bandwidth or fewer bytes helps; more FLOP/s doesn't.

Each step reads the weights once and uses them for every sequence; only each sequence's KV cache is extra. On our test laptop's Apple GPU, a step took 6.3 ms for one sequence and 9.3 ms for 8, so throughput rose from 159 to 860 tokens/s.

Limits: the longer step slows each user, and every conversation's KV cache needs memory.

Weights plus KV cache per conversation times conversations must fit. Weights take about 26 GB. Each cache is 2 × 40 layers × 5,120 × 2,048 tokens × 2 bytes = 1.68 GB, so the remaining 54 GB holds 32, matching AI Engineering's example. Real servers fit fewer: activations, the framework and fragmentation take memory too.

The cache is linear in tokens and bytes per number, so cap conversation length or store the cache in fewer bytes.

Continuous batching. Static batching waits for a full batch and dynamic caps the wait, but both finish a batch together, so a short reply waits for its longest neighbour. Continuous batching, Orca's iteration-level scheduling, returns each reply when done and refills its seat.

Simulated from our measured Apple GPU steps (200 requests, at most 8 at once), continuous gave 343 tokens/s and a 22 ms median first token; static batches gave 299 tokens/s and a 5.3 s median wait.

Write the three costs down, then compare times.

# d model, f hidden (often 4d), T tokens, bf16
flops = 2 * (2 * T * d * f)      # two matmuls
w_bytes = 2 * d * f * 2          # both weights
act_bytes = T * (2 * d + f) * 2  # in, out, hidden
# tensor parallel over n GPUs: one all-reduce of T*d
comm = 2 * (n - 1) / n * T * d * 2   # bytes per GPU
t_compute = flops / n / peak_flops
t_hbm = (w_bytes / n + act_bytes) / hbm_bw
t_net = comm / link_bw

The largest time binds. With few tokens, as in decode, intensity is roughly T FLOPs per byte, below a GPU's FLOPs-to-bandwidth ratio, so HBM binds; big prefill batches turn compute-bound. Sharding divides compute and weights by n but not the all-reduce, so over slow inter-node links the network binds.

A compiler turns the model graph into fewer, faster kernels. Kernel fusion matters most: chaining elementwise ops, norms or attention into one kernel keeps intermediates in registers or shared memory instead of round-tripping HBM, and most such ops are memory-bound. Others: constant folding, layout changes to what tensor cores want, lowering to vendor libraries, autotuning tile sizes per GPU, static memory planning, and CUDA graphs against launch overhead. Fitting the hardware means sizing tiles to SM resources and keeping matmuls in tensor-core shapes and precisions.

In eager PyTorch, each op dispatches to a kernel, often from cuBLAS or cuDNN, and the CPU launches it asynchronously on a CUDA stream. The GPU runs it as thread blocks across its SMs, reading HBM. A compiler like torch.compile instead captures a graph, fuses ops and generates Triton kernels; CUDA graphs replay launches with less CPU overhead. Small ops end up launch- or memory-bound, large matmuls compute-bound on tensor cores.

First prove it: a profiler shows each step waiting on the next batch, and GPU utilisation dips in a sawtooth. A quick test is to repeat one cached batch; if throughput jumps, the input pipeline is the bottleneck.

Then move work out of the hot path. Tokenise, filter and dedupe offline, writing packed, pre-tokenised sequences into large shards instead of many small files. At train time, stream shards with parallel workers, prefetching and pinned memory, shuffle by shard order plus a buffer, and make the loader resumable from a saved position.

Compare each kernel's arithmetic intensity with the GPU's ratio of compute to bandwidth. Large matmuls are compute-bound; softmax, norms, elementwise ops and small-batch decode are memory-bound, so fuse them.

Naive attention writes the full sequence-by-sequence score matrix to HBM: quadratic memory and traffic. FlashAttention tiles it through on-chip SRAM instead.

MFU counts only the model's required FLOPs against peak; HFU also counts recomputation, so activation checkpointing raises HFU, not MFU.

Profile: timeline gaps for data loading, launch overhead and communication; per-kernel achieved bandwidth versus tensor-core utilization.

Parameters × bits per weight ÷ 8. GPT-2 small's 124,439,808 weights take 497.8 MB in FP32 and 62.2 MB in INT4, an eighth, plus small per-group scales. Decode reads every weight each step, so bytes per weight set both the memory and the bytes each step moves.

They split the bits differently: FP16 has 5 exponent and 10 fraction bits, BF16 8 and 7. Exponent bits buy range, fraction bits precision, so FP16 is finer but tops out at 65,504, while BF16 keeps FP32's range. Llama 2 shipped in BF16; teams loading it in FP16 found quality much worse than advertised.

Divide each value by the group's largest magnitude, multiply by 127, round; store the integers plus one scale. A value moves at most half a step.

One outlier widens the step for its whole group: adding an invented value 20 times our largest GPT-2 weight took the step from 0.00271 to 0.0542, and the largest error on the real weights from 0.00126 to 0.027. Hence per-block scales.

Post-training, weight-only quantization. I didn't train the model, PTQ is the common path frameworks offer in a few lines, and weight quantization affects quality more stably.

It buys less than the byte ratio suggests. For Llama 2 13B at 2,048 tokens with a 16-bit cache, INT4 weights take 6.5 GB instead of 26, so 43 conversations fit instead of 32: the cache takes most of the memory. Next lever: shrink the cache. Hands-On LLMs' rule of thumb: stay at 4 bits or above.

It's a ceiling, not a forecast. Batch-1 decode is bandwidth-bound, so time should track bytes read, but the KV cache doesn't shrink. Scaling our measured 9.1 ms CPU step by bytes gives about 1.3 ms at INT4: an estimate, not a measurement, with the cache now 13% of the bytes.

And the format needs hardware and kernel support: format conversion costs compute, and LLM.int8() claims only modest matrix-multiply speedups. I'd measure step time on our setup.

import numpy as np

def quantize(x, qmin=-128, qmax=127):
    lo, hi = min(x.min(), 0.0), max(x.max(), 0.0)
    scale = (hi - lo) / (qmax - qmin) or 1.0
    zp = int(np.clip(round(qmin - lo / scale),
                     qmin, qmax))
    q = np.round(x / scale) + zp
    return np.clip(q, qmin, qmax).astype(np.int8), scale, zp

def dequantize(q, scale, zp):
    return (q.astype(np.float32) - zp) * scale

Asymmetric quantization maps the observed range onto -128 to 127 with a scale and a zero point, keeping 0.0 exactly representable. Interviewers check rounding before adding the zero point, clipping, the all-zero case, and outliers: one huge value stretches the scale and crushes the rest, so clip to a percentile or use per-channel scales.

Quantization-aware training simulates low-precision rounding during fine-tuning, so accuracy holds better than post-training quantization. Distillation trains a small student on a teacher's soft outputs. Eval mode disables dropout and makes batch norm use running statistics; forgetting it corrupts metrics. Contrastive learning pulls matching pairs together in embedding space and pushes others apart. Verify each on a fixed held-out set and the target hardware: accuracy per slice, latency and memory versus the baseline.

The KV cache works within one request, so each decode step computes only the new token's keys and values: after a 960-token prompt on our laptop CPU, 9.6 ms a step, against 228.8 ms without. A prompt cache keeps a shared opening's keys and values across requests. An answer cache skips the model entirely.

A prompt cache reuses only an identical prefix, and the match stops at the first differing token: here, the very first. It isn't a provider quirk: a token's keys and values depend on every token before it.

Order prompts static first: fixed instructions and tools, then shared reference material, then per-request parts like the date, retrieved snippets and the question.

When writes aren't followed by reads. Anthropic charges 1.25 times the base input price for a 5-minute cache write and 0.1 times for a read (as of 2026-10-21). A 2,000-token prefix sent once costs 24% more than not caching. Rare traffic is the same: the entry expires, so every request pays to write it again. Shared by 10 requests, a 10,000-token prefix saves 77%.

Mostly not. A prefix cache removes the prefix's prefill, so it cuts TTFT and input cost, not decode. In our CPU estimate with 128 shared and 64 unique tokens and 400 output tokens, the first token came about 48% sooner but the reply finished only about 1% sooner: 3.32 s to 3.30 s. That estimate is optimistic, too: it ignores attention to the cached prefix and loading the cache.

To finish sooner, change decode: shorter outputs, faster hardware, or fewer bytes per weight.

An answer cache serves a reply written for an earlier request. Risks: stale answers to time-sensitive queries; leaks, like the book's return-policy answer built from user X's details and served to user Y; and, with semantic matching, wrong hits. Embeddings capture topic more than truth: with all-MiniLM-L6-v2, "The Earth orbits the Sun." and its reverse scored 0.982, while the best true paraphrase scored 0.645.

I'd measure hit rate and wrong-hit rate on our own queries, and keep personal queries out.

The provider keeps the processed prefix, the keys and values prefill computed for your system prompt, and reuses them when the next request starts with the same tokens. You skip that prefill: cheaper input and a faster first token.

It only matches an exact prefix, so put static text first and anything that changes, like dates or user data, after it.

Input tokens × input price plus output tokens × output price. Output tokens cost more, roughly two to four times input across providers, per AI Engineering. At $2 and $10 per million, Sonnet 5.5's list prices (checked 21 October 2026), an illustrative 2,000-in, 500-out request spends 56% of its cost on the reply.

Divide the price per hour by tokens per hour: the book's $2 an hour at 100 tokens/s is about $5.56 per million output tokens. Batching raises throughput, so it cuts cost: at an illustrative $2 an hour, our Apple GPU's measured throughput gives $3.50 per million at batch 1 and $0.26 at batch 64, about 13 times cheaper. But bigger batches slow each user, and the KV cache caps the batch.

B. Track cost per useful answer: spend divided by answers that were on time and correct. Retrying until success takes 1 ÷ p attempts on average, so A costs $0.004 ÷ 0.5 = $0.008 per correct answer and B $0.006 ÷ 0.95 ≈ $0.0063. Lateness counts too: in lesson 1's CPU run, 83 of 120 requests met the SLO, so each on-time request cost 1.45 times the average.

Break-even requests per day = machine cost per day ÷ API cost per request, if one machine can serve that many. An illustrative $48-a-day machine against $0.002304 per request crosses at about 20,800 a day.

Left out: idle time (at 1,000 requests a day that machine costs $0.048 per request, 0.23% busy); quality, since our throughput is GPT-2 on a laptop, not a frontier model; engineering, which the book says can cost more than the API; and tokenizers, so compare per request. Re-run it as traffic grows.

Find where the tokens go first: output tokens cost more each, but long prompts can still dominate the bill.

Then cap and shorten outputs, cache the static prefix, trim retrieved context, route easy requests to a smaller model, run offline work through batch APIs, and cache whole answers only for exact repeats. Track cost per correct answer, so savings don't hide a quality drop.

When it passes your own eval set at your quality bar, not a leaderboard. Narrow, well-specified tasks like classification, extraction and routing often do; open-ended reasoning often doesn't. Compare cost per correct answer, include the cost of hosting it yourself, and consider a router: the small model first, escalating hard cases.

Route by difficulty and latency budget: easy, high-volume requests to the small model, hard ones to the large. The router can be rules, a small classifier, or a cascade that tries the cheap model and escalates when a check fails, which adds a second call's latency on escalations.

Set thresholds on a labelled set by plotting quality against cost per request, and watch p95, where cascades hurt.

Route on what each query needs. The on-device model takes device actions, short answers and private requests: fast, offline, no per-query cost. Open-ended, knowledge-heavy or long reasoning requests go to the server model.

The router is rules for clear intents, then a small on-device classifier scoring difficulty, plus escalation when the local answer's confidence is low. Ask before sending personal context off the device. Track quality per cost and latency: share escalated, quality against always-server on a labelled set, and tail latency. Retrain on misrouted cases.

Clients submit a job with an idempotency key; the API stores it durably and returns a job id, the same id for any retry. Jobs queue per tenant and priority, with quotas against starvation.

Workers resolve the prompt version pinned in the request, route to a model by task and budget, and store results; clients poll or get a signed webhook. Control cost with token estimates at admission, budget caps, batch APIs for non-urgent work, prompt caching and per-tenant metering. Failures back off, then dead-letter.

Everything except the thing you changed: model and precision, prompt and reply lengths, batch, concurrency and arrival rate, warm-up, repeats, environment and unit. Report percentiles, TTFT and TPOT separately.

Lengths alone move results: on our laptop CPU, the first token took 12.8 ms after 16 prompt tokens and 223.3 ms after 960.

Not on that evidence. Three runs of the same CPU prefill on our laptop, same model, versions, threads and prompt, gave 219.6, 223.3 and 239.4 ms: 9% apart. Their methods also differed, so no cause could be named. A 5% gap sits inside that spread. Interleave several runs of each version with an identical method, and compare the spreads.

Ask: for which phase, prompt length and batch? On our test laptop the Apple GPU's first-token advantage was 1.6× at 16 tokens and 2.3× at 960, but batch-1 decode only about 1.4×, which fits bandwidth-bound decode with similar copy ceilings. Short prompts with long replies are almost all decode, so the headline barely applies.

The book's three questions: can it run the workload, how long does it take, what does it cost. I'd measure my own prefill and decode mix, comparing ratios from one run.

Measure it myself: their number is their workload on their day. Latency depends on my prompts and sampling settings, so I'd send my own prompts at my lengths and concurrency and measure p90 TTFT, as the book's criteria table does on an internal prompt dataset.

Test at peak concurrency, since some providers throttle. Check quality too: the same Llama models scored differently across providers in a Cerebras experiment. Compare cost per request, not per token.

It has no headroom. Capacity, the highest arrival rate still within 300 ms at p90, is 6 per second against a target of 9: eight seats fill as traffic grows, and requests queue. A larger batch limit buys capacity until the KV caches break the 2 GB budget.

It's simulated from measured steps, but those were measured at a 128-token context while this workload reaches 768, so it's optimistic. First real measurement: p90 TTFT and capacity under the real workload.

Perplexity is exp of the mean negative log-likelihood per token, the model's effective branching factor on held-out text. Being per token, it isn't comparable across tokenizers: longer tokens mean fewer, harder predictions. Normalise to bits per byte to compare. It also measures fit to a text distribution, not helpfulness, correctness or instruction following, so it says little about a chat model's quality, and it shifts with the eval set.

Benchmark the workload you'll serve: realistic prompt and output length distributions, arrival rates and concurrency, not one fixed prompt. Report time to first token, inter-token latency and end-to-end at p50 and p99, plus throughput at a latency SLO; peak throughput at unbounded latency means little.

Change one knob at a time, batch size, precision, tensor-parallel degree, engine, and rerun a quality eval after each, since quantization can quietly cost accuracy. Warm up, pin versions, repeat runs, and profile to explain each result.

Ask which mistake costs more. Precision when a false alarm is expensive, like an agent that issues refunds or bans accounts. Recall when a miss is expensive, like a safety agent that must catch harmful content. F1 when both matter and positives are rare, so accuracy misleads. With known costs, weight errors directly.

Agreement tells you whether labels are reliable enough to score against. Raw agreement overstates it, since raters agree by chance; Cohen's kappa corrects for that with two raters, and Krippendorff's alpha handles many raters and missing labels. Low agreement often means a vague rubric.

LLM judges are cheap but biased by position, length and self-preference. Let humans build the gold set; trust a judge once its agreement with them matches human-to-human agreement.

First remove the randomness you control: greedy decoding or a fixed seed, and a pinned model, prompt and parameters. Even then, GPU serving isn't bitwise deterministic, because batch composition changes floating-point reduction order, so cache outputs for exact replay.

For the score, treat runs as samples. Measure the standard deviation across repeats in a pilot, then pick n so the standard error, sd over root n, is well below the difference you need to detect.

Vary what drives risk: actor types and behaviours, speeds and gaps, occlusions, road geometry, weather and sensor noise. Sample from real-world distributions fitted to driving logs, so results mean something. Collisions are rare, so plain Monte Carlo wastes runs; use importance sampling to oversample dangerous regions, then weight each run by its likelihood ratio, keeping the collision-rate estimate unbiased, with confidence intervals. Validate the simulator against logged events.

One token sequence. The chat template writes each role as marker tokens around its content; AI Engineering: system and user prompts are "processed the same way".

The system prompt can count for more only because it comes first, and because post-training may teach the model to favour it.

From the template: 9 special markers, 3 role names and 3 blank lines. The text was counted with cl100k_base standing in for Llama 3's tokenizer, so treat it as close, not exact; each marker is exactly one token. Short chats are mostly template. You can't remove those tokens, but every token is billed and prefilled on each call, so make the rest cheap to repeat: static first.

Suspect the chat template: its errors "cause silent failures". Diff the token IDs of one fixed prompt before and after; the printed string can't catch it. In the lab's two-message chat, an extra newline keeps 25 tokens but changes token #4; markers tokenized as text give 72 tokens, none special, and print identically; no assistant header gives 21. Fix: use the template that ships with the model.

As one layer, not a guarantee. Wallace et al. rank system over user over third-party content; the email is tool output, the lowest of the book's four levels. Misaligned lower-level instructions should be ignored; aligned ones, like "use a formal tone", still followed. It's trained behaviour: on GPT-3.5 it improved defence against system-prompt extraction by 63%, better, not perfect. The template must cooperate too: content encoded as plain text, so a typed <|eot_id|> can't close a turn.

Make it specific: the scale and what to do when unsure, "integer scores only" if it returns 4.5, the format with no preamble, and an end marker. A persona changes the standard: in the book's illustration, the same essay "might" get 2 out of 5 plainly and 4 from a first-grade teacher. Add or remove one component at a time and judge the effect. Then cut tokens: in our transcription of the book's Table 5-2, arrow-style examples take 26 tokens against 37.

Learning a task from examples in the prompt, with no change to the weights; each example is a shot. Nothing is learned between calls, so every shot is input tokens on every request; the context window caps how many fit. For GPT-4, in Microsoft's 2023 analysis, few-shot gave only limited gains over zero-shot.

JSON mode guarantees valid JSON, not its content. Output can still be cut off at the output-token limit, which doesn't parse. And parseable outputs can still carry wrong types ("2" for an integer), invented labels, missing or extra fields, or values past a limit. Only a schema check catches those.

The model may lean towards the label it saw last. Zhao et al. traced unstable few-shot accuracy to biases, including towards answers near the end of the prompt; Lu et al. found order alone can swing between near state-of-the-art and random-guess performance. Both studied GPT-3-era models, so test yours: cover every label and the hard cases, shuffle, compare a few orders on a fixed test set, and end on a marker.

Repair fixes syntax, never content. On the lab's ten hand-written, illustrative outputs, repair lifts parsing from 50% to 90%, but still only 50% are valid: the wrong type, invented label, missing field and long summary were valid JSON all along, and a cut-off summary can't be finished.

Content failures need a retry with the validator's error, capped, then a logged fallback.

The arithmetic is right only if failures are independent: 1 − 0.1³ = 99.9% valid, for 1.11 calls on average. That's a best case. The book notes a model "tends to repeat similar mistakes", so a retry on the same input may fail the same way. I'd measure how often retries actually succeed on our failing inputs, and report parse, valid and correct rates, with and without repair, so a parser fix can't hide a prompt problem.

AI Engineering lists five places: prompting, post-processing and test time compute are "more like bandages"; constrained sampling and finetuning are the intensive treatment. LinkedIn's defensive parser took correct YAML from 90% to 99.99%, as the book cites, but post-processing works only when mistakes are easy to fix. For fixed labels, constrain sampling to them, or use a classifier head. Neither makes the label right: GPT-2 constrained to planet names picks " Earth" over " Jupiter". Still measure the correct rate on labelled examples.

Zero-shot gives only the instruction; few-shot adds worked examples of input and output. Add them when the format or judgement is hard to describe but easy to show. Each example costs tokens on every request and the model copies its quirks, so pick varied examples and measure that they help.

Don't parse generated text; read the next-token distribution after the prompt.

import torch

def p_yes(logits, yes_ids, no_ids):
    # logits: (V,) next-token logits
    lp = logits.log_softmax(-1)
    ly = torch.logsumexp(lp[yes_ids], -1)
    ln = torch.logsumexp(lp[no_ids], -1)
    return torch.sigmoid(ly - ln)

Each label has several token spellings, so combine each group with log-sum-exp, then renormalise over the two groups: a sigmoid of the difference. Threshold it after calibrating on labelled data, since raw probabilities are often skewed, and check token ids, because a leading space changes them.

Don't ask the model for Word markup. Have it emit structured output, JSON with only allowed blocks such as heading with level, paragraph, list and table, each naming a permitted style, enforced by schema-constrained decoding.

A deterministic renderer then builds the document from the template's styles. Validate rules the schema can't express, like heading order, and retry with the error. Evaluate by diffing rendered styles against the spec.

Template the prompt with the schema and one example, and use the API's JSON or structured-output mode if it has one. Parse the reply and validate it with Pydantic against the same schema. On failure, retry with the validation error, a few times at most, then fail loudly.

Fitting isn't helping. Every input token is billed and prefilled on every request, and a long context isn't necessarily used well. In the lesson's help centre, all 14 articles are 299 tokens against 25 for the one that answers, about 12 times the input; with thousands of documents, the ratio grows.

Fusion rewards agreement, not correctness: each retriever gives 1 / (k + rank), k = 60 being typical.

For "how do I stop paying every month", BM25 returned only the Wi-Fi article, through the word "every"; embeddings ranked it third. It scored 1/61 + 1/63 and beat the cancellation article's 1/61, so the answer, ranked #2, was cut. Judge selection on queries with known answers, and leave k room.

It can. In Liu et al., GPT-3.5-Turbo with 20 passages scored 75.8% with the answer first, 53.8% at position 10, below its 56.1% closed-book score, and 63.2% last. Claude-1.3 was much flatter, and the best end varied by model. These are 2023 models, so sort by relevance (best first, or best next to the question) and compare both on a needle-style test.

Every method loses something; you choose what. Dropping the oldest turns is simple, but the book warns it "can be fatally wrong": in an illustrative 40-turn chat, keeping the last 10 loses turn 1, which account and plan the user is cancelling. A model-written summary keeps more, but costs a call and is new text, so it can omit or misstate details. Truncating, or routing to a larger-window model, are the blunt options. Write down what must survive (goal, IDs, numbers, constraints) and test it early, late and buried.

Retrieval scores similarity, not whether a passage answers. For "is this toy safe for a two-year-old", BM25, embeddings and their fusion all put "safe for children over three" above "not safe for children under three", the one that answers; embeddings barely register "not". With top 1 the model gets only the misleading passage. Evaluate retrieval against questions with known answers, keep more than one candidate, and rerank with a more precise second stage.

Prompt engineering is wording the instruction. Context engineering decides everything else in the window on each call: retrieved passages, history, tool results, examples, and their order.

The window is a budget, so the work is selecting what's relevant, compressing what's long, and placing what matters where the model attends best.

Treat the context window as a budget. For long documents, chunk along their structure and index chunks plus document summaries, so retrieval goes coarse to fine. For long queries, have a model rewrite them into focused sub-queries, retrieve for each, then merge and rerank.

Pack the best chunks up to a token budget, with the most relevant at the start or end, since models use the middle of long contexts worst. When the answer needs a whole document, use map-reduce: extract per chunk, then answer over the extracts.

First find which cost hurts: attention compute grows quadratically with length and dominates long prefills, while the KV cache grows linearly and caps decode batch size.

Exact fixes first. FlashAttention tiles the work in on-chip memory, never storing the n × n score matrix: same result, memory linear in n. For the cache: grouped-query attention, quantisation, paged allocation.

Then approximations that change the model: sliding-window or sparse patterns like Longformer's local plus global attention, or linear-attention and state-space layers. Past one GPU, ring attention splits the sequence across devices. Re-check long-context quality.

Keep two stores per user: a small profile of stable facts and preferences, always in the prompt, and episode summaries retrieved by relevance.

After each conversation, a model extracts atomic facts with a timestamp and source; newer facts override conflicting ones, and duplicates merge. At query time, fit the profile plus the top retrieved memories into a fixed token budget.

The hard part is trust: users can see and delete memories, sensitive data needs consent and retention limits, stores are isolated per user, and remembered text is data, never instructions. Measure whether it helps.

RAG and fine-tuning fill different gaps: RAG supplies changing or citable facts, fine-tuning shapes format, style or a narrow skill. Over the window: retrieve only the relevant chunks, or summarise in a map-reduce pass. Conversation state: keep recent turns verbatim, summarise older ones, and store durable facts outside the prompt. Prompts: version them like code, with templates and a regression suite. Treat retrieved text as data, never instructions, to limit prompt injection.

def load_history(fetch_page):
    msgs, cursor = [], None
    while True:
        page = fetch_page(cursor)
        msgs.extend(page["messages"])
        cursor = page.get("next_cursor")
        if not cursor:
            return msgs

def compact(msgs, limit, count, summarize, keep=6):
    if sum(count(m) for m in msgs) <= limit:
        return msgs
    while True:
        cut = max(len(msgs) - keep, 0)
        note = {"role": "system",
                "content": summarize(msgs[:cut])}
        out = [note] + msgs[cut:]
        if sum(count(m) for m in out) <= limit:
            return out
        if keep == 0:
            raise ValueError("summary exceeds limit")
        keep //= 2

Follow the cursor until none comes back, retrying failed pages. Compact only when over the limit: summarise older turns into one note, keep the recent turns verbatim, and halve that window if it still doesn't fit. Count with the model's own tokenizer, and never separate a tool call from its result.