Loading…

Skip to content

Attention head

Also called attention heads

One independent attention computation. Models run many heads in parallel, each free to look for a different relationship.

Learn it in the lessonMany heads, real weightsModel Foundations · Attention: How Tokens Use Their Context

Taught alongside

Attention
Query, key and value
Causal mask
Softmax
Transformer block
Residual connection
Layer norm
Feed-forward network (MLP)

Home
Learn
Interview
Playground

Try

Jump to a lesson

Loading…

↑ ↓ move↵ openesc close