Glossary
83 terms from the lessons. In lesson text, hover any dotted-underlined term for its definition.
A
- Arithmetic intensity
- Operations done per byte read from memory. Low intensity means bandwidth is the limit; high intensity means compute is.
- Where this is taught →
- Attention
- The mechanism that lets each token take in information from other tokens, weighted by how relevant they score.
- Where this is taught →
- Attention head
- One independent attention computation. Models run many heads in parallel, each free to look for a different relationship.
- Where this is taught →
- Autoregressive generation
- Writing one token at a time, appending each to the input before predicting the next.
- Where this is taught →
B
- Batch API
- An inference API that returns results later (often hours) in exchange for a lower price, for work that doesn't need an answer straight away.
- Where this is taught →
- BF16 (bfloat16)
- A 16-bit float with 1 sign, 8 exponent and 7 fraction bits. It has the same range as FP32 but less precision than FP16.
- Where this is taught →
- BM25
- A keyword-search scoring function that ranks documents by the query words they contain, weighting rare words more and long documents less.
- Where this is taught →
- Byte pair encoding (BPE)
- A way to learn a subword vocabulary: start from single symbols and repeatedly merge the most frequent adjacent pair. Encoding replays the learned merges in order.
- Where this is taught →
C
- Causal mask
- In GPT-style models, the rule that a token can only attend to itself and earlier tokens, never later ones.
- Where this is taught →
- Chat template
- The model-specific rule that turns a list of role-tagged messages into the one token sequence the model was fine-tuned on, with special markers around each turn. A wrong template fails silently.
- Where this is taught →
- Compute-bound
- Work whose time is set by how much arithmetic it needs; a chip with more operations per second speeds it up.
- Where this is taught →
- Constrained sampling
- At each generation step, masking every token that would break a rule, grammar or schema, then renormalising the remaining probabilities before sampling. Guarantees format, not correctness.
- Where this is taught →
- Context window
- The maximum number of tokens a model can process at once: prompt and output together.
- Where this is taught →
- Contextual embedding
- A token's vector after the transformer layers have mixed in the surrounding text, so the same word gets different vectors in different sentences.
- Where this is taught →
- Continuous batching
- Batching that re-decides the batch before every decode step: finished replies leave at once and waiting requests take their seats.
- Where this is taught →
- Cosine similarity
- How closely two vectors point the same way, from −1 to 1, ignoring their lengths. The usual way to compare embeddings.
- Where this is taught →
- Cost per request
- What one request costs end to end: input tokens × input price + output tokens × output price for an API, or machine time × hourly price when self-hosting. Comparable across models with different tokenizers, unlike a price per token.
- Where this is taught →
- Cost per useful answer
- Total spend divided by answers that were both on time (met the SLO) and correct. Retries, late replies and wrong answers all raise it.
- Where this is taught →
D
- Dot product
- Multiply two vectors element by element and add the results. Grows with both their alignment and their lengths.
- Where this is taught →
E
- Embedding
- A vector of numbers that represents a token or a text, arranged so that similar meanings end up close together.
- Where this is taught →
- Embedding table
- The learned matrix with one row per vocabulary entry. Embedding a token ID means reading its row.
- Where this is taught →
- Exact caching
- Storing whole answers and returning one only when exactly the same query is asked again.
- Where this is taught →
- Exact match
- A scorer that passes an output only if it equals the expected answer exactly. Suits short, exact answers; fails correct answers that are formatted differently.
- Where this is taught →
F
- Feed-forward network (MLP)
- The part of a transformer block that processes each token's vector on its own, after attention has mixed information between tokens.
- Where this is taught →
G
- Goodput
- The rate of requests that meet the latency objective (SLO). Slow requests count toward throughput but not goodput.
- Where this is taught →
- Greedy decoding
- Always picking the single most likely next token. Deterministic, but prone to repetition.
- Where this is taught →
- Grounding
- Giving the model relevant, trusted text in its context so its answer can be based on that text rather than memory alone.
- Where this is taught →
H
- Hallucination
- Fluent, confident output that is false or not supported by the input.
- Where this is taught →
I
- In-context learning
- Teaching a model a task through instructions and examples in the prompt, with no change to its weights. Each example is a shot: zero-shot uses none, few-shot several.
- Where this is taught →
- Indirect prompt injection
- Prompt injection delivered through data the model reads later, such as a web page, email, document or tool result, rather than typed by the user.
- Where this is taught →
- Instruction hierarchy
- A trained priority order for conflicting instructions (Wallace et al., 2024): system over user over model and tool output. Lower-level instructions that conflict with higher ones are ignored; ones that agree are followed.
- Where this is taught →
- Inter-token latency
- The gap between two consecutive output tokens. Its average is the time per output token.
- Where this is taught →
J
- Jailbreaking
- Getting a model past its safety features. OWASP treats it as a form of prompt injection.
- Where this is taught →
K
- KV cache
- Stored keys and values of earlier tokens, so each decode step only computes them for the new token. Exact, but costs memory.
- Where this is taught →
L
- Layer norm
- Rescales each token's vector to a standard spread before a sub-layer, which keeps training stable.
- Where this is taught →
- LM head
- The final layer that turns the last position's vector into one logit per vocabulary token.
- Where this is taught →
- Logit
- A raw score the model gives each vocabulary token for being next. Softmax turns logits into probabilities.
- Where this is taught →
- Logprob
- The natural log of a token's probability. Useful for debugging, but not a measure of whether an answer is true.
- Where this is taught →
- Lost in the middle
- The finding (Liu et al., 2023) that language models often use information at the start or end of a long context better than information in the middle, giving a U-shaped accuracy curve.
- Where this is taught →
M
- Memory bandwidth-bound
- Work whose time is set by how fast data moves from memory to the arithmetic units; more memory bandwidth, or fewer bytes to move, speeds it up.
- Where this is taught →
- MLPerf Inference
- MLCommons' public inference benchmark. Its rules fix the model, dataset and accuracy target; the Server scenario reports the highest Poisson arrival rate that meets the latency limits.
- Where this is taught →
- Model bandwidth utilization (MBU)
- The share of peak memory bandwidth inference uses: parameters × bytes per parameter × tokens per second, over the peak, for one sequence.
- Where this is taught →
N
- Needle in a haystack
- A long-context test: insert a piece of information at different positions in a long prompt and check whether the model can find and use it.
- Where this is taught →
P
- Percentile (p50, p99)
- pN is the value N% of measurements fall at or below: p50 is the median, p99 is beaten by all but the slowest 1%. Used for latency because the mean hides the tail.
- Where this is taught →
- Position embedding
- A vector added to each token's embedding so the model knows where the token sits in the sequence.
- Where this is taught →
- Post-training quantization
- Quantizing a model after it has been fully trained. It is the most common approach, and the one application developers use.
- Where this is taught →
- Prefill
- Processing the whole prompt in one parallel pass before the first output token can be produced.
- Where this is taught →
- Prompt caching
- Keeping the KV cache of a prompt's shared opening, such as a system prompt or long document, so later requests that start with the identical prefix skip prefilling it.
- Where this is taught →
- Prompt injection
- Text that gets a model to follow instructions its developer never intended, possible because instructions and data reach the model as one token stream.
- Where this is taught →
- Prompt version
- One exact, named revision of a prompt. It is logged with every output it produces so results can be traced and compared; any edit, even a typo fix, makes a new version.
- Where this is taught →
Q
- Quantization
- Storing a model's numbers in a format with fewer bits, such as FP16, INT8 or INT4, to cut memory and the bytes each step reads. Strictly it means converting to integers, but in practice it covers any precision reduction.
- Where this is taught →
- Quantization-aware training
- Simulating low precision during training so the model learns to work well in it. It does not make training faster.
- Where this is taught →
- Query, key and value
- Three learned projections of each token's vector. A query is compared with every key to get attention weights; the weights then mix the values.
- Where this is taught →
R
- Reciprocal rank fusion
- Combining several retrievers' rankings by summing 1 / (k + rank) for each document, with k typically 60.
- Where this is taught →
- Residual connection
- Adding a sub-layer's output back onto its input, so each block refines the token vectors instead of replacing them.
- Where this is taught →
- Roofline
- A chart of achievable operations per second against arithmetic intensity: a sloped roof set by memory bandwidth and a flat roof set by peak compute, meeting at the ridge.
- Where this is taught →
S
- Seed
- The starting value of a random number generator. The same seed and settings reproduce the same samples on the same setup.
- Where this is taught →
- Semantic caching
- Returning a stored answer when a new query's embedding is similar enough to a stored query's, above a threshold. Raises the hit rate, but can return the answer to a different question.
- Where this is taught →
- Semantic search
- Finding documents by comparing embeddings of the query and the documents, so matches can share meaning without sharing words.
- Where this is taught →
- Service level objective (SLO)
- A target a service promises to meet, such as 'first token within 100 ms and at most 10 ms per token after that'.
- Where this is taught →
- Softmax
- Turns a list of scores into probabilities: exponentiate each score, then divide by the total so they sum to 1.
- Where this is taught →
- Special token
- A vocabulary entry with a control meaning rather than text, such as end-of-text or a chat-role marker. User text must never become one.
- Where this is taught →
- Structured output
- Model output in a fixed machine-readable format, such as JSON with known keys or one label from a list, so code can parse and use it.
- Where this is taught →
- Subword
- A token that can be a whole word or a fragment of one. Common words get one token; rare words are built from several.
- Where this is taught →
- System prompt
- The developer's instructions, sent as the first message with the system role. The model gets it in the same sequence as the user's text; it carries more weight only through position and training.
- Where this is taught →
T
- Temperature
- A number that divides the logits before softmax: below 1 sharpens the distribution, above 1 flattens it, 0 means greedy.
- Where this is taught →
- Test set
- A fixed collection of inputs with known expected answers, run unchanged against every version so their results can be compared case by case.
- Where this is taught →
- Throughput
- Work done per second across all requests, usually output tokens per second. Not the same as how fast any one user's reply arrives.
- Where this is taught →
- Time per output token (TPOT)
- The average time between output tokens after the first one: how fast a streamed reply flows.
- Where this is taught →
- Time to first token (TTFT)
- How long from sending a request until the first output token arrives. Mostly the prefill, so it grows with the prompt's length.
- Where this is taught →
- Time to publish
- Time until the first token the user can see. Later than time to first token when the model first generates hidden steps such as a plan.
- Where this is taught →
- Token
- A piece of text from a model's fixed vocabulary: a word, part of a word, a character or a byte sequence. Models read and write tokens, not characters.
- Where this is taught →
- Token ID
- The integer a tokenizer assigns to a token: its position in the vocabulary. IDs only mean something within the vocabulary that produced them.
- Where this is taught →
- Tokenizer
- The component that cuts text into tokens and maps each to an integer ID, and turns IDs back into text.
- Where this is taught →
- Tool calling
- A model emits a structured request (a tool name and JSON arguments) instead of text; the application checks and runs the function, then returns the result into the model's context.
- Where this is taught →
- Top-k
- Sampling only among the k most likely tokens, after renormalising their probabilities.
- Where this is taught →
- Top-p (nucleus sampling)
- Sampling only among the smallest set of most-likely tokens whose probabilities add up to at least p.
- Where this is taught →
- Transformer block
- The repeated unit of a transformer: attention and a feed-forward network, each wrapped in a layer norm and a residual connection.
- Where this is taught →
- Trust boundary
- The line between context written by the developer or the user being served, and context anyone else controls, which must not be allowed to give orders.
- Where this is taught →
U
- Utilization
- The share of a resource's capacity actually in use. A rented machine that sits idle still bills by the hour, so low utilization raises the cost per request.
- Where this is taught →
V
- Vocabulary
- The fixed list of every token a tokenizer knows. Its size is a design choice made before training.
- Where this is taught →
W
- Warm-up
- Untimed runs before a benchmark starts timing, so one-off costs such as loading and kernel compilation don't land in the results.
- Where this is taught →