Chapter 3
Unicode and Text Normalization
Code points, bytes, grapheme clusters, UTF-8, normalization, case folding, punctuation, and scripts without spaces.
What this chapter does
Text is not just English words separated by spaces. This chapter teaches enough real text handling to avoid brittle tokenization.
Lessons
Read these in order.
Start with the chapter introduction, then move through the topic lessons. The order is chosen so each page can reuse ideas from the pages before it.
- 01Introduction
Why real text handling matters before tokenization begins.
- 02Code Points, Bytes, and Graphemes
Three layers of text representation that are easy to confuse.
- 03UTF-8
A variable-length byte encoding that lets Unicode text live in files and network streams.
- 04Normalization and Case Folding
Making equivalent-looking text comparable without erasing meaning carelessly.
- 05Punctuation and Whitespace
Visible and invisible marks that affect boundaries, meaning, and token counts.
- 06Scripts Without Spaces
Why word boundaries are not equally visible across writing systems.
Review and practice
Close the chapter deliberately.
Use the conclusion and revision notes before the chapter exercises. Hints and solutions are collected here, while lesson-level exercises reveal their own help inline.
What Unicode and normalization establish before tokens and vocabularies.
Summary and Revision NotesA compact review of bytes, code points, graphemes, UTF-8, normalization, and whitespace.
ExercisesChapter-level practice for Unicode layers and text normalization choices.
HintsLow-spoiler nudges for the Chapter 3 exercises.
SolutionsExplained solutions for the Chapter 3 exercises.
Before moving on
- Distinguish code points, bytes, and visible characters.
- Explain why simple splitting fails on real text.
- Inspect text before tokenizing it.
Where this leads
- Tokens and Vocabularies
- Subword Tokenization