Embedding Matrix Review

An embedding matrix has one row per token id.

If the vocabulary has 10,000 tokens and the embedding dimension is 256, the matrix shape is:

10000 x 256

Looking up token id 37 selects row 37. That row is the vector passed to the model.

The token id is discrete. The embedding vector is continuous and trainable.

Exercise

A vocabulary has 5,000 tokens. The embedding dimension is 100. How many rows are in the embedding matrix?