Conclusion

Tokenization defines the model's prediction units.

You saw:

• a token is a model-facing unit
• a vocabulary is the known token set
• unknown tokens lose identity
• character tokens provide coverage but lengthen sequences
• word tokens shorten sequences but create vocabulary pressure
• rare words motivate subword tokenization

The next chapter introduces subwords, especially byte-pair encoding, as a practical compromise between characters and words.