Chapter 4
Tokens and Vocabularies
Tokens, vocabularies, unknown tokens, character tokens, word tokens, subword motivation, rare words, and vocabulary size tradeoffs.
What this chapter does
Tokenization is a modeling decision. This chapter compares character and word views before subwords enter.
Lessons
Read these in order.
Start with the chapter introduction, then move through the topic lessons. The order is chosen so each page can reuse ideas from the pages before it.
- 01Introduction
Tokens as the interface between raw text and language-model computation.
- 02What Is a Token?
A token as the unit the model reads, predicts, and stores in a vocabulary.
- 03Vocabularies and Unknowns
How a finite token list creates ids, counts, and unknown-token behavior.
- 04Character Tokens
A small-vocabulary view that makes spelling visible but stretches sequences.
- 05Word Tokens
A short-sequence view that makes words visible but struggles with rare and new forms.
- 06Rare Words and Vocabulary Size
The tradeoff between vocabulary size, sequence length, memory, and open vocabulary.
Review and practice
Close the chapter deliberately.
Use the conclusion and revision notes before the chapter exercises. Hints and solutions are collected here, while lesson-level exercises reveal their own help inline.
What character and word tokenization establish before subword tokenization.
Summary and Revision NotesA compact review of tokens, vocabularies, unknowns, character tokens, word tokens, and vocabulary tradeoffs.
ExercisesChapter-level practice for token choices, vocabularies, and rare-word tradeoffs.
HintsLow-spoiler nudges for the Chapter 4 exercises.
SolutionsExplained solutions for the Chapter 4 exercises.
Before moving on
- Explain why neither characters nor words solve everything.
- Compare sequence length and vocabulary size.
- Understand why subwords are needed.
Where this leads
- Subword Tokenization
- Count-Based Language Models