LLM Primer

Introduction

the same text can become different token sequencestext: language modelscharacter viewl a n g u a g e ...longer sequence, small vocabularyword viewlanguage | modelsshorter sequence, larger vocabularytokenization is a modeling choice, not a neutral cleanup step
Characters avoid unknown words; words preserve larger chunks. Subwords will later balance the two.

A language model does not predict raw visual text. It predicts tokens.

A token is the unit the model sees and predicts. A vocabulary is the set of tokens the model knows how to represent.

Tokenization is therefore not a harmless formatting step. It defines the units of the prediction problem.

In this chapter, you will compare:

  • character tokens
  • word tokens
  • unknown tokens
  • vocabulary size
  • rare-word behavior

Subwords come next. First we need to see why characters and words each solve one problem while creating another.