Chapter 9
Embeddings in Language
Embedding matrices, distributional intuition, co-occurrence, word2vec context, learned similarity, nearest neighbors, analogies, polysemy, and static versus contextual representations.
What this chapter does
Embeddings connect token identities to learned vectors, but language soon exposes the limits of one fixed vector per token.
Lessons
Read these in order.
Start with the chapter introduction, then move through the topic lessons. The order is chosen so each page can reuse ideas from the pages before it.
- 01Introduction
Token ids become learned vectors before entering neural language models.
- 02Embedding Matrix Review
One trainable row per token id, selected by lookup.
- 03Distributional Intuition
How similar contexts can shape related token vectors.
- 04Word2Vec as History
A historical precursor showing prediction pressure can produce useful word geometry.
- 05Learned Similarity and Neighbors
Reading nearest neighbors as clues about learned representation geometry.
- 06Polysemy and Context
Why one static vector struggles with words that have multiple uses.
Review and practice
Close the chapter deliberately.
Use the conclusion and revision notes before the chapter exercises. Hints and solutions are collected here, while lesson-level exercises reveal their own help inline.
What embeddings establish before neural language models.
Summary and Revision NotesA compact review of embedding lookup, distributional intuition, Word2Vec, nearest neighbors, and polysemy.
ExercisesChapter-level practice for embedding shapes and representation interpretation.
HintsLow-spoiler nudges for the Chapter 9 exercises.
SolutionsExplained solutions for the Chapter 9 exercises.
Before moving on
- Explain how token embeddings are learned.
- Inspect a small embedding table.
- Explain why fixed token vectors are not enough for all language use.
Where this leads
- Neural Language Models
- Recurrent Neural Networks