Word Tokens
Word tokenization uses larger units.
For:
language models learn
the word tokens are:
language | models | learn
The advantage is shorter sequences. Common words remain intact, and local meaning can be easier to inspect.
The cost is vocabulary size. A word-level model needs entries for many words, names, variants, typos, and domain terms. A new word can become unknown unless the vocabulary already contains it.
Word tokenization also depends on word boundaries. As the previous chapter showed, spaces are not universal word separators.
LM-C04-T04-001Exercise: Word sequence length
If a sentence has 6 space-separated words, how many word tokens does this simple tokenizer produce?
Compute it first, then check your number.
Words shorten sequences but make the vocabulary problem larger.