Summary and Revision Notes
Core idea
Text has layers. A language model eventually sees tokens, but tokenization sits on top of bytes, Unicode, visible characters, normalization, whitespace, and script conventions.
Terms
| Term | Meaning |
|---|---|
| byte | storage unit |
| code point | Unicode number for a character |
| grapheme cluster | visible user-perceived character |
| UTF-8 | common encoding of Unicode as bytes |
| normalization | making selected text forms consistent |
Caution
Do not assume:
- byte count equals visible character count
- lowercasing is always harmless
- punctuation is always noise
- spaces define words in every script
Before moving on
You should be able to explain why a tokenizer should be tested on more than plain English words separated by spaces.