Embedding table
The learned matrix with one row per vocabulary entry. Embedding a token ID means reading its row.
Interview questions about Embedding table
- What is an embedding table, and what does it mean for two embeddings to be similar?
It's a learned matrix with one row per vocabulary entry, and embedding a token just reads its row.
- An input embedding table is too large for one device. Compare sharding it by vocabulary with sharding it by hidden dimension.
By vocabulary, each device owns a range of rows, looks up the tokens it owns, writes zeros for the rest, and an all-reduce sum assembles full vectors.