Autoregressive generation
Also called autoregressive
Writing one token at a time, appending each to the input before predicting the next.
Interview questions about Autoregressive generation
- Cover the fundamentals: self- versus cross-attention, autoregressive decoding and sampling, where RL fits, and how you evaluate the result.
Self-attention draws queries, keys and values from one sequence; cross-attention takes queries from the decoder and keys and values from another source, like…
- Design and implement a KV cache for autoregressive decoding in a Transformer: tensor shapes for prefill and per-step decode, per-layer caches, batching, memory limits and beam search.
- Design an image generation system end to end: data curation, diffusion versus autoregressive architectures, training objectives, compute planning and evaluation.
Data: large sets of image-text pairs filtered for quality, safety, duplicates and licensing, recaptioned by a vision-language model, since detailed captions…