Summary and Revision Notes
Core idea
Tokenization chooses the units a language model sees and predicts.
Comparison
| View | Strength | Cost |
|---|---|---|
| character tokens | coverage | long sequences |
| word tokens | short common sequences | large vocabulary and unknowns |
| subword tokens | compromise | extra tokenizer machinery |
Vocabulary
The vocabulary is the set of tokens represented directly. In a neural model, the token vocabulary usually corresponds to rows in an embedding table and columns in the output distribution.
Before moving on
You should be able to explain why neither pure characters nor pure words are a perfect tokenization strategy for modern language models.