Introduction

A token id is just an integer. It has no meaning by itself.

An embedding turns that id into a learned vector. The vector can then enter the same neural-network machinery studied in Deep Learning.

In language, embeddings are powerful because they let tokens share geometry. Words or subwords used in similar contexts can move toward related regions of the embedding space.

What this chapter covers

• embedding matrix review;
• distributional intuition;
• word2vec as history;
• learned similarity and nearest neighbors;
• polysemy and context.