Normalization, Whitespace, and Punctuation
Text often contains variation that may or may not matter.
Hello!
hello
hello
These strings are visibly related, but they are not identical. A preparation step might lowercase them, trim whitespace, or keep punctuation.
Normalization means making some text forms more consistent. Common choices include:
- lowercasing
- trimming repeated spaces
- replacing unusual whitespace
- standardizing punctuation
- leaving the text unchanged
There is no universal correct choice. Lowercasing can help a small model share
evidence between Apple and apple, but it may also erase a distinction
between a company and a fruit. Removing punctuation can simplify counting, but
punctuation often carries meaning.
If Tea, tea, and TEA are lowercased before counting, how many times does tea appear?
Compute it first, then check your number.
Do not treat cleanup as harmless by default. Every cleanup rule changes what the model can see.