Embedding Matrix

An embedding matrix has one row per item and one column per vector coordinate.

shape = vocabulary_size x embedding_dimension

If there are 10,000 tokens and each token has a 256-dimensional vector, the matrix has:

10,000 x 256 = 2,560,000

trainable numbers.

embedding matrix: vocabulary size by embedding dimensionid 0id 1id 2id 3id 4d1d2d3d4d5d6each row is a learned vector; each column is one coordinate
The embedding matrix stores one trainable vector per discrete item.

The embedding dimension is a design choice. A larger dimension gives the model more room to represent differences, but it also adds parameters and can overfit when data is limited.

The rows are not independent facts. They are adjusted by training pressure from the task.

Exercise: Matrix shape

An embedding table has 50 items and embedding dimension 4. How many trainable numbers does it contain?

Compute it first, then check your number.

Exercise: Row count

If a vocabulary has 8 items, how many rows does its embedding matrix have?

Compute it first, then check your number.