Vocabularies and Unknowns

A vocabulary is the set of tokens a model can represent directly.

For a tiny vocabulary:

{I, like, tea, coffee}

the sentence:

I like water

contains one unknown word: water.

Older word-level systems often used a special unknown token:

I | like | <unk>

That keeps the sequence representable, but it loses the identity of water.

This is one reason modern language models usually avoid pure word vocabularies. Rare words, names, typos, code identifiers, and new terms appear constantly.

Unknown handling is a modeling decision, not an afterthought.