Character Versus Word Views

The same text can be inspected at different levels.

text: tea
characters: t | e | a
words: tea

A character view makes spelling visible. A word view makes larger chunks visible.

For the sentence:

I like tea

the word view has 3 items:

I | like | tea

The character view has more items, especially if spaces are counted:

I |   | l | i | k | e |   | t | e | a

Neither view is always best. Character models avoid unknown words, but their sequences become long. Word models shorten common text, but rare words become a problem.

Tokenization will later generalize this choice. For now, just notice that a model's units are chosen, not discovered automatically.