Summary and Revision Notes

Key ideas

• An embedding is a learned vector for a discrete item.
• A lookup selects a row from an embedding table.
• An embedding matrix has shape vocabulary_size x embedding_dimension.
• Discrete ids are usually addresses, not meaningful magnitudes.
• Embeddings are parameters updated by gradients.
• Representation geometry helps inspect learned vectors.
• Similarity after learning can reveal useful structure, but it is evidence rather than proof.
• Token ids become vectors through embedding lookup.

Common formulas

embedding_parameters = vocabulary_size * embedding_dimension
e_next = e - learning_rate * gradient
cosine_similarity(a, b) = dot(a, b) / (||a|| ||b||)

Common mistakes

• Treating token ids as meaningful numbers.
• Thinking embeddings are fixed dictionary definitions.
• Assuming each coordinate has a simple human label.
• Reading nearest neighbors without checking data and task context.
• Forgetting that embedding rows are trainable parameters.

Before moving on

You should be able to explain an embedding lookup, compute the size of an embedding matrix, update a small embedding row, and describe why embeddings matter for language models.