Rare Words and Open Vocabulary

A language model cannot store a separate word token for every possible word. People create new names, misspell words, combine words, write code, and mix languages. A fixed word vocabulary will always meet text it has not seen.

This is the open-vocabulary problem.

Character tokenization avoids unknown words because every word can be spelled from characters. But it pays with longer sequences. Word tokenization gives shorter sequences, but it needs an unknown token for words outside the vocabulary.

Subword tokenization tries to keep both useful properties:

• common words can stay as one token;
• rare words can be built from smaller pieces;
• unseen words can often be represented without becoming a single unknown.

Example

Suppose a vocabulary contains:

low, lower, est, slow, s

The word slowest can be represented as:

s low est

That representation is not perfect English morphology. It is a useful computational compromise.

Exercise

If unhappiness is split as un happy ness, how many tokens are produced?