Conclusion

Subword tokenization is the practical bridge between character and word views. It keeps frequent pieces compact while still allowing rare or new forms to be represented.

The main lesson is that tokenization is part of modeling. It determines the sequence the model reads, the vocabulary the model predicts from, and the length of the contexts that must be handled.

Next we make prediction concrete with count-based language models.