LLM Primer

Introduction

visible text has several representation layersbytesstorage unitscode pointsUnicode numbersgraphemesvisible characterstokensmodel unitstokenization should happen after text representation is understood
Text is not only a string of English words; each layer affects what the model sees.

Text looks simple on screen, but a program does not store "visible words" in a single obvious way.

Before tokenization, we need a small amount of text representation literacy:

  • bytes store data
  • Unicode code points name characters
  • grapheme clusters describe visible user-perceived characters
  • normalization can make different encodings comparable
  • scripts and punctuation do not all behave like English

This chapter is not a Unicode reference manual. It teaches enough to prevent a bad assumption: that text is just English words separated by spaces.