Summary and Revision Notes

Core ideas

• Word vocabularies cannot cover every possible word.
• Character vocabularies avoid unknown words but create longer sequences.
• Subword tokenization is a compromise between those extremes.
• BPE repeatedly merges frequent adjacent pairs.
• A BPE tokenizer is defined by an ordered merge list.
• WordPiece and unigram tokenization are related subword families.
• Special tokens, padding, and truncation are part of the model interface.

Check yourself

• Can you explain why unhappiness might be split into pieces?
• Can you trace two BPE merges by hand?
• Can you explain why padding tokens should not count as ordinary text?
• Can you explain why truncation can change model behavior?