Contextual embedding
Also called contextual embeddings
A token's vector after the transformer layers have mixed in the surrounding text, so the same word gets different vectors in different sentences.
Interview questions about Contextual embedding
- Anthropic reports large gains from contextual retrieval. What exactly did they measure, and what does it cost?
1 − recall@20: the share of relevant chunks missing from the top 20.