Rare Words and Vocabulary Size
Vocabulary size is a tradeoff.
A small vocabulary is easier to store and learn, but it creates more unknowns or longer spell-out sequences.
A large vocabulary can preserve more words directly, but it uses more parameters and still cannot include every future name, typo, code identifier, or new term.
Suppose a word-level vocabulary has:
10,000 tokens
If each token needs one embedding vector, then the embedding table has one row per token. Doubling the vocabulary roughly doubles the number of embedding rows.
This is the pressure that motivates subwords. A tokenizer can represent common chunks directly while still breaking rare words into smaller pieces.
If a vocabulary has 100 tokens, how many rows does its embedding table need?
Compute it first, then check your number.
The next chapter studies this compromise directly.