Character Tokens

Character tokenization uses small units.

For:

cat

the character tokens are:

c | a | t

The advantage is coverage. If the character set is broad enough, new words can still be represented as sequences of characters.

The cost is length. A sentence that has 8 word tokens may have 40 or more character tokens. Longer sequences are harder to model, especially when relationships span many positions.

Character tokenization also makes useful chunks less explicit. The model must learn that c, a, and t together form a familiar unit.

Characters solve unknown words by making the sequence longer.