Feed-forward network (MLP)
Also called feed-forward network, MLP
The part of a transformer block that processes each token's vector on its own, after attention has mixed information between tokens.
Interview questions about Feed-forward network (MLP)
- Design a streaming system that tokenizes arriving text, embeds it, trains a classifier and serves low-latency predictions.
Text arrives on a log. Stream workers tokenize with the tokenizer pinned to the encoder version, embed in GPU micro-batches, and store vectors with ID…
- What's inside one transformer block, and where do most of its parameters sit?
Two sub-blocks, each layer norm, then the sublayer, then a residual add: first multi-head attention, then an MLP.
- A pre-norm transformer block trains badly. What do you check around the norms and residuals?
Pre-norm should read x + attn(norm(x)), then x + mlp(norm(x)): normalise each sublayer's input and add its output to the un-normalised stream.
- With no starter code, implement a GPT-style transformer decoder from scratch in about 30 minutes.
- Implement scaled dot-product multi-head attention and a Transformer block from scratch, including the backward pass for attention, without autograd.
Have the forward pass cache what the backward pass needs: the softmax probabilities P.
- Explain transformer and mixture-of-experts fundamentals, including how training differs from inference.
A transformer stacks blocks of attention, which mixes tokens, and an MLP, each with a residual.
- How does a vision encoder feed images into a multimodal LLM, and where are the bottlenecks?
A vision encoder, usually a ViT trained contrastively like CLIP, cuts the image into patches and embeds each one.