Chapter 5
Subword Tokenization
Rare words, open vocabulary, BPE, merge operations, WordPiece, unigram tokenization, byte-level BPE, special tokens, padding, and truncation.
What this chapter does
Subword tokenization is the practical compromise between characters and words used by many modern language models.
Lessons
Read these in order.
Start with the chapter introduction, then move through the topic lessons. The order is chosen so each page can reuse ideas from the pages before it.
- 01Introduction
Subwords as the practical compromise between character and word tokenization.
- 02Rare Words and Open Vocabulary
Why fixed word vocabularies cannot cover all names, spellings, code, and new forms.
- 03Byte Pair Encoding
Starting from small units and repeatedly merging frequent adjacent pairs.
- 04Merge Operations
How an ordered merge list turns characters into larger token pieces.
- 05WordPiece and Unigram Tokenization
Related subword-tokenization families and the shared idea behind them.
- 06Special Tokens, Padding, and Truncation
The non-text tokens and length decisions that form the model interface.
Review and practice
Close the chapter deliberately.
Use the conclusion and revision notes before the chapter exercises. Hints and solutions are collected here, while lesson-level exercises reveal their own help inline.
What subword tokenization establishes before count-based language models.
Summary and Revision NotesA compact review of open vocabulary, BPE, related tokenizers, special tokens, padding, and truncation.
ExercisesChapter-level practice for subword splits, BPE merges, padding, and vocabulary size.
HintsLow-spoiler nudges for the Chapter 5 exercises.
SolutionsExplained solutions for the Chapter 5 exercises.
Before moving on
- Trace a tiny BPE merge example.
- Explain why subword tokenization helps LLMs.
- Recognize tokenizer choices as model-interface choices.
Where this leads
- Count-Based Language Models
- Transformers