What Is a Token?
A token is a model-facing unit.
For the text:
language models
a character tokenizer might produce:
l | a | n | g | u | a | g | e | | m | o | d | e | l | s
a word tokenizer might produce:
language | models
Both are tokenizations. They produce different sequences for the same text.
The model predicts token IDs, not the informal idea of "a word." Later, token IDs are mapped to vectors and processed by neural networks.
LM-C04-T01-001Exercise: Word tokens
If language models learn is split on spaces, how many word tokens are produced?
Compute it first, then check your number.
The token is the unit of prediction. Choose it carefully.