Conclusion

Text representation affects language modeling.

You saw that:

• bytes are storage units
• UTF-8 encodes Unicode text as bytes
• code points are not always what readers perceive as one character
• normalization and case folding merge chosen forms
• punctuation and whitespace can carry meaning
• scripts do not all follow English spacing habits

The next chapter turns this representation knowledge into the first modeling interface: tokens and vocabularies.