Mid-level interview question

What does the causal mask do, and what does it imply for a decoder model reading 'The ladder did not reach the window because it was too short'?

Q: What does the causal mask do, and what does it imply for a decoder model reading 'The ladder did not reach the window because it was too short'?

It hides later tokens: before softmax, every score whose key comes after the query becomes minus infinity, so those weights are exactly 0 and the rest still sum to 1. So a decoder disambiguates only from the past: whether the ladder sentence ends "too short" or "too high", GPT-2 small's attention rows up to and including " it" were identical in all 12 layers and 12 heads. That also makes the KV cache exact.

Model answer

It hides later tokens: before softmax, every score whose key comes after the query becomes minus infinity, so those weights are exactly 0 and the rest still sum to 1.

The causal maskStep 1 / 4

Scroll sideways to see the whole diagram

Rows are the token looking (its query), columns the token looked at (its key). Each cell is a score: the query's dot product with the key, divided by √d.

Text version

Rows are the token looking (its query), columns the token looked at (its key). Each cell is a score: the query's dot product with the key, divided by √d.
The causal mask: before the softmax, every score where the key comes after the query is set to minus infinity.
Softmax turns those cells into exactly 0, and the remaining weights in each row still add up to 1.
Token 1 sees only itself, token 2 sees tokens 1 and 2, and so on. Adding tokens later never changes an earlier row, which is why the KV cache is exact.

The causal mask as taught in lesson 3, on the attention lab's three tokens “The river bank”.

So a decoder disambiguates only from the past: whether the ladder sentence ends "too short" or "too high", GPT-2 small's attention rows up to and including " it" were identical in all 12 layers and 12 heads. That also makes the KV cache exact.

Practise the rest

20 Attention questions, 492 in all, each with a model answer. Free.

Read the lesson Practise Attention

Mid-level interview question

What does the causal mask do, and what does it imply for a decoder model reading 'The ladder did not reach the window because it was too short'?

Model answer

It hides later tokens: before softmax, every score whose key comes after the query becomes minus infinity, so those weights are exactly 0 and the rest still sum to 1.

The causal maskStep 1 / 4

Scroll sideways to see the whole diagram

Rows are the token looking (its query), columns the token looked at (its key). Each cell is a score: the query's dot product with the key, divided by √d.

Text version

Rows are the token looking (its query), columns the token looked at (its key). Each cell is a score: the query's dot product with the key, divided by √d.
The causal mask: before the softmax, every score where the key comes after the query is set to minus infinity.
Softmax turns those cells into exactly 0, and the remaining weights in each row still add up to 1.
Token 1 sees only itself, token 2 sees tokens 1 and 2, and so on. Adding tokens later never changes an earlier row, which is why the KV cache is exact.

The causal mask as taught in lesson 3, on the attention lab's three tokens “The river bank”.

Practise the rest

20 Attention questions, 492 in all, each with a model answer. Free.

Read the lesson Practise Attention

Model answer

More Attention interview questions

Model answer

More Attention interview questions