Normalization and Case Folding
Unicode normalization makes equivalent text forms more consistent.
Case folding is a stronger form of lowercasing used for text comparison. It can make matching easier, but it also changes the original text.
For language modeling, normalization is a choice:
- it can reduce accidental variation
- it can merge forms that should remain distinct
- it can change what the model is asked to predict
For example, lowercasing:
Apple -> apple
apple -> apple
may be useful in a tiny fruit corpus, but harmful if Apple refers to a
company and apple refers to fruit.
LM-C03-T03-001Exercise: Case folding effect
If Apple and apple are lowercased before counting, how many distinct lowercase forms remain?
Compute it first, then check your number.
Normalization should be recorded as part of the experiment, not hidden as a minor cleanup detail.